ref consistency check infra setup
PATCH v8 8/9 fsck: add ref name check for files backend
In reply to shejialuoThe git-fsck(1) only implicitly checks the reference, it does not fully check refs with bad format name such as standalone "@" and name ending with ".lock".
In order to provide such checks, add a new fsck message id "badRefName" with default ERROR type. Use existing "check_refname_format" to explicit check the ref name. And add a new unit test to verify the functionality.
--- Documentation/fsck-msgids.txt | 3 + fsck.h | 1 + refs/files-backend.c | 20 +++++++ t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++ 4 files changed, 125 insertions(+) create mode 100755 t/t0602-reffiles-fsck.sh
Documentation/fsck-msgids.txt +3 -0
@@ -19,6 +19,9 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefName`:: + (ERROR) A ref has a bad name. + `badTagName`:: (INFO) A tag has an invalid format.
fsck.h +1 -0
@@ -31,6 +31,7 @@ enum fsck_msg_type { FUNC(BAD_NAME, ERROR) \ FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ + FUNC(BAD_REF_NAME, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
refs/files-backend.c +20 -0
@@ -3419,6 +3419,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,const char *refs_check_dir, struct dir_iterator *iter); +static int files_fsck_refs_name (struct fsck_options *o, + const char *gitdir UNUSED, +const char *refs_check_dir, +struct dir_iterator *iter) +{ +struct strbuf sb =STRBUF_INIT ; +int ret =0 ; + +if (check_refname_format (iter->basename,REFNAME_ALLOW_ONELEVEL )) { +strbuf_addf (&sb,"%s/%s" , refs_check_dir, iter->relative_path); + ret =fsck_refs_report (o,NULL , sb.buf, +FSCK_MSG_BAD_REF_NAME , +"invalid refname format" ); + } + +strbuf_release (&sb); +return ret; +} + static int files_fsck_refs_dir(struct ref_store *ref_store, struct fsck_options *o, const char *refs_check_dir,
@@ -3470,6 +3489,7 @@ static int files_fsck_refs(struct ref_store *ref_store, {int ret;files_fsck_refs_fn fsck_refs_fns[]= { + files_fsck_refs_name,NULL };
t/t0602-reffiles-fsck.sh new file +101 -0
101 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v8 9/9 fsck: add ref content check for files backend
In reply to shejialuoEnhance the git-fsck(1) command by adding a check for reference content in the files backend. The new functionality ensures that symrefs, real symbolic link and regular refs are validated correctly.
In order to check the trailing content of the regular refs, add a new parameter trailing to parse_loose_ref_contents.
For symrefs, parse_loose_ref_contents will set the "referent". However, symbolic link could be either absolute or relative. Use "strbuf_add_real_path" to read the symbolic link and convert the relative path to absolute path. Then use "skip_prefix" to make it align with symref "referent".
Thus, the symrefs and symbolic links could share the same interface. Add a new function "files_fsck_symref_target" which aims at checking the following things:
1. whether the pointee is under the refs/ directory.
2. whether the pointee name is correct.
3. whether the pointee path is a wrong type in filesystem.
Last, add the following FSCK MESSAGEs:
1. "badRefContent(ERROR)": A ref has a bad content
2. "badSymrefPointee(ERROR)": The pointee of a symref is bad.
3. "trailingRefContent(WARN)": A ref content has trailing contents.
--- Documentation/fsck-msgids.txt | 9 +++ fsck.h | 3 + refs.c | 2 +- refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++- refs/refs-internal.h | 5 +- t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++ 6 files changed, 269 insertions(+), 5 deletions(-)
Documentation/fsck-msgids.txt +9 -0
@@ -19,9 +19,15 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefContent`:: + (ERROR) A ref has a bad content. + `badRefName`:: (ERROR) A ref has a bad name. +`badSymrefPointee`:: + (ERROR) The pointee of a symref is bad. + `badTagName`:: (INFO) A tag has an invalid format.
@@ -167,6 +173,9 @@ `nullSha1`:: (WARN) Tree contains entries pointing to a null sha1. +`trailingRefContent`:: + (WARN) A ref content has trailing contents. + `treeNotSorted`:: (ERROR) A tree is not properly sorted.
fsck.h +3 -0
@@ -32,6 +32,8 @@ enum fsck_msg_type { FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ FUNC(BAD_REF_NAME, ERROR) \ + FUNC(BAD_REF_CONTENT, ERROR) \ + FUNC(BAD_SYMREF_POINTEE, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
@@ -72,6 +74,7 @@ enum fsck_msg_type { FUNC(HAS_DOTDOT, WARN) \ FUNC(HAS_DOTGIT, WARN) \ FUNC(NULL_SHA1, WARN) \ + FUNC(TRAILING_REF_CONTENT, WARN) \ FUNC(ZERO_PADDED_FILEMODE, WARN) \ FUNC(NUL_IN_COMMIT, WARN) \FUNC (LARGE_PATHNAME ,WARN ) \
refs.c +1 -1
@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store, } result =parse_loose_ref_contents (content.buf, oid, referent, type, - failure_errno); + failure_errno,NULL ); done:strbuf_release (&full_path);
refs/files-backend.c +143 -2
181 lines not on this page · raw message
refs/refs-internal.h +3 -2
@@ -709,11 +709,12 @@ struct ref_store {/* * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for -* invalid contents. +* invalid contents. Also *trailing is set to the first character after the +* refname or NULL if the referent is not empty. */ int parse_loose_ref_contents (const char *buf,struct object_id *oid,struct strbuf *referent,unsigned int *type, -int *failure_errno); +int *failure_errno,const char **trailing);/* * Fill in the generic part of refs and add it to our collection of
t/t0602-reffiles-fsck.sh +110 -0
114 lines not on this page · raw message
Signature · 1 line
2.45.2
v9 cover ref consistency check infra setup
In reply to shejialuoHi All:
This version changes the following things:
1. Instead of using "oid_skiplist", use "skip_oids".
2. In "fsck_refs_error_function", I define a static struct strbuf here, the reason why I define it to be static is that we may call this function many times. It's a bad idea to allocate memory too frequently. And we should never free its memory. I made a mistake here, I should call strbuf_reset function every time. So this version, add "strbuf_reset" to make the behavior correct.
shejialuo (9):
fsck: rename "skiplist" to "skip_oids"
fsck: add a unified interface for reporting fsck messages
fsck: add refs-related options and error report function
refs: set up ref consistency check infrastructure builtin/refs: add verify subcommand builtin/fsck: add git-refs verify child process
files-backend: add unified interface for refs scanning
fsck: add ref name check for files backend
fsck: add ref content check for files backend
Documentation/fsck-msgids.txt | 12 ++ Documentation/git-refs.txt | 13 ++ builtin/fsck.c | 32 ++++- builtin/mktag.c | 1 + builtin/refs.c | 44 ++++++ fsck.c | 108 +++++++++++--- fsck.h | 63 ++++++--- object-file.c | 11 +- refs.c | 7 +- refs.h | 8 ++ refs/debug.c | 11 ++ refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++- refs/packed-backend.c | 8 ++ refs/refs-internal.h | 11 +- refs/reftable-backend.c | 8 ++ t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++ 16 files changed, 746 insertions(+), 57 deletions(-) create mode 100755 t/t0602-reffiles-fsck.sh
Range-diff against v8:
1: 61e475840f ! 1: e044f933de fsck: rename "skiplist" to "oid_skiplist"
@@ Metadata
Author: shejialuo <[email protected]>
## Commit message ##
- fsck: rename "skiplist" to "oid_skiplist"
+ fsck: rename "skiplist" to "skip_oids"
The "skiplist" field in "fsck_options" is related to objects. Because we
are going to introduce ref consistency check, the "skiplist" name is too
@@ Commit message
"skiplist" here. However, the type for "skiplist" is `struct oidset`
which is totally unsuitable for refs.
- To avoid above ambiguity, rename "skiplist" to "oid_skiplist".
+ To avoid above ambiguity, rename "skiplist" to "skip_oids".
Mentored-by: Patrick Steinhardt <[email protected]>
Mentored-by: Karthik Nayak <[email protected]>
@@ fsck.c: void fsck_set_msg_types(struct fsck_options *options, const char *values
if (equal == len)
die("skiplist requires a path");
- oidset_parse_file(&options->skiplist, buf + equal + 1,
-+ oidset_parse_file(&options->oid_skiplist, buf + equal + 1,
++ oidset_parse_file(&options->skip_oids, buf + equal + 1,
the_repository->hash_algo);
buf += len + 1;
continue;
@@ fsck.c: void fsck_set_msg_types(struct fsck_options *options, const char *values
const struct object_id *oid)
{
- return opts && oid && oidset_contains(&opts->skiplist, oid);
-+ return opts && oid && oidset_contains(&opts->oid_skiplist, oid);
++ return opts && oid && oidset_contains(&opts->skip_oids, oid);
}
__attribute__((format (printf, 5, 6)))
@@ fsck.h: struct fsck_options {
unsigned strict:1;
enum fsck_msg_type *msg_type;
- struct oidset skiplist;
-+ struct oidset oid_skiplist;
++ struct oidset skip_oids;
struct oidset gitmodules_found;
struct oidset gitmodules_done;
struct oidset gitattributes_found;
+@@ fsck.h: struct fsck_options {
+ };
+
+ #define FSCK_OPTIONS_DEFAULT { \
+- .skiplist = OIDSET_INIT, \
++ .skip_oids = OIDSET_INIT, \
+ .gitmodules_found = OIDSET_INIT, \
+ .gitmodules_done = OIDSET_INIT, \
+ .gitattributes_found = OIDSET_INIT, \
2: f2576d88a9 ! 2: daaf3d0ffe fsck: add a unified interface for reporting fsck messages
@@ builtin/mktag.c: static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;
## fsck.c ##
@@ fsck.c: static int object_on_skiplist(struct fsck_options *opts,
- return opts && oid && oidset_contains(&opts->oid_skiplist, oid);
+ return opts && oid && oidset_contains(&opts->skip_oids, oid);
}
-__attribute__((format (printf, 5, 6)))
@@ fsck.h: int is_valid_msg_type(const char *msg_id, const char *msg_type);
struct fsck_options {
fsck_walk_func walk;
@@ fsck.h: struct fsck_options {
- };
-
- #define FSCK_OPTIONS_DEFAULT { \
-- .skiplist = OIDSET_INIT, \
-+ .oid_skiplist = OIDSET_INIT, \
- .gitmodules_found = OIDSET_INIT, \
.gitmodules_done = OIDSET_INIT, \
.gitattributes_found = OIDSET_INIT, \
.gitattributes_done = OIDSET_INIT, \
3: c3c2dda50c ! 3: 40da85ae30 fsck: add refs-related options and error report function
@@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,
+{
+ static struct strbuf sb = STRBUF_INIT;
+
++ strbuf_reset(&sb);
+ strbuf_addstr(&sb, checked_ref_name);
+ if (oid)
+ strbuf_addf(&sb, " -> (%s)", oid_to_hex(oid));
@@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *
unsigned strict:1;
+ unsigned verbose_refs:1;
enum fsck_msg_type *msg_type;
- struct oidset oid_skiplist;
+ struct oidset skip_oids;
struct oidset gitmodules_found;
@@ fsck.h: struct fsck_options {
.gitattributes_done = OIDSET_INIT, \
4: e826dc17ec = 4: a38ea1b117 refs: set up ref consistency check infrastructure
5: 33cac4882b ! 5: 8320f56e0b builtin/refs: add verify subcommand
@@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c
+ * Explicitly free the allocated array and "oid_skiplist"
+ */
+ free(fsck_refs_options.msg_type);
-+ oidset_clear(&fsck_refs_options.oid_skiplist);
++ oidset_clear(&fsck_refs_options.skip_oids);
+ return ret;
+}
+
6: 32668e3543 = 6: 6614a06ef5 builtin/fsck: add `git-refs verify` child process
7: df83b2a990 = 7: 928cc96396 files-backend: add unified interface for refs scanning
8: c696c15651 = 8: 4d50d4932f fsck: add ref name check for files backend
9: 8b0f3aeb9c = 9: 7edb810819 fsck: add ref content check for files backendSignature · 1 line
2.45.2
PATCH v9 1/9 fsck: rename "skiplist" to "skip_oids"
The "skiplist" field in "fsck_options" is related to objects. Because we are going to introduce ref consistency check, the "skiplist" name is too general which will make the caller think "skiplist" is related to both the refs and objects.
It may seem that for both refs and objects, we should provide a general "skiplist" here. However, the type for "skiplist" is struct oidset which is totally unsuitable for refs.
To avoid above ambiguity, rename "skiplist" to "skip_oids".
--- fsck.c | 4 ++-- fsck.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
fsck.c +2 -2
@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)if (!strcmp (buf,"skiplist" )) {if (equal == len)die ("skiplist requires a path" ); -oidset_parse_file (&options->skiplist, buf + equal +1 , +oidset_parse_file (&options->skip_oids, buf + equal +1 , the_repository->hash_algo); buf += len +1 ;continue ;
@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)static int object_on_skiplist (struct fsck_options *opts,const struct object_id *oid) { -return opts && oid &&oidset_contains (&opts->skiplist, oid); +return opts && oid &&oidset_contains (&opts->skip_oids, oid); } __attribute__((format (printf, 5, 6)))
fsck.h +2 -2
@@ -136,7 +136,7 @@ struct fsck_options {fsck_error error_func; unsigned strict:1;enum fsck_msg_type *msg_type; -struct oidset skiplist; +struct oidset skip_oids;struct oidset gitmodules_found;struct oidset gitmodules_done;struct oidset gitattributes_found;
@@ -145,7 +145,7 @@ struct fsck_options { };#define FSCK_OPTIONS_DEFAULT { \ - .skiplist = OIDSET_INIT, \ + .skip_oids = OIDSET_INIT, \ .gitmodules_found = OIDSET_INIT, \ .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \
Signature · 1 line
2.45.2
PATCH v9 2/9 fsck: add a unified interface for reporting fsck messages
In reply to shejialuoThe static function "report" provided by "fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "vfsck_report" by adding "checked_ref_name" parameter following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.
Like "report", the "vfsck_report" function will use "error_func" registered in "fsck_options" to report customized messages. Change "error_func" prototype to align with the new "vfsck_report".
Then, change "report" function to use "vfsck_report" to report objects related messages. Add a new function called "fsck_refs_report" to use "vfsck_report" to report refs related messages.
--- builtin/fsck.c | 15 ++++----- builtin/mktag.c | 1 + fsck.c | 81 ++++++++++++++++++++++++++++++++++++------------- fsck.h | 40 +++++++++++++++--------- object-file.c | 11 ++++--- 5 files changed, 101 insertions(+), 47 deletions(-)
builtin/fsck.c +8 -7
@@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)return -1 ; } -static int fsck_error_func(struct fsck_options *o UNUSED, -const struct object_id *oid, -enum object_type object_type, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +static int fsck_objects_error_func(struct fsck_options *o UNUSED, +const struct object_id *oid, +enum object_type object_type, + const char *checked_ref_name UNUSED, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {switch (msg_type) {case FSCK_WARN :
@@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) fsck_walk_options.walk = mark_object; fsck_obj_options.walk = mark_used; - fsck_obj_options.error_func = fsck_error_func; + fsck_obj_options.error_func = fsck_objects_error_func;if (check_strict) fsck_obj_options.strict =1 ;
builtin/mktag.c +1 -0
@@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT; static int mktag_fsck_error_func(struct fsck_options *o UNUSED, const struct object_id *oid UNUSED, enum object_type object_type UNUSED, + const char *checked_ref_name UNUSED,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message)
fsck.c +60 -21
115 lines not on this page · raw message
fsck.h +26 -14
@@ -114,22 +114,27 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);typedef int (*fsck_walk_func )(struct object *obj,enum object_type object_type,void *data,struct fsck_options *options); -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */ +/* +* callback function for reporting errors when checking either objects or refs +*/ typedef int (*fsck_error )(struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const char *checked_ref_name,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message); -int fsck_error_function (struct fsck_options *o, -const struct object_id *oid,enum object_type object_type, -enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, -const char *message); -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message); +int fsck_objects_error_function (struct fsck_options *o, +const struct object_id *oid,enum object_type object_type, +const char *checked_ref_name, +enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, +const char *message); +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +const struct object_id *oid, +enum object_type object_type, +const char *checked_ref_name, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);struct fsck_options {fsck_walk_func walk;
@@ -150,7 +155,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function \ + .error_func = fsck_objects_error_function \ }#define FSCK_OPTIONS_STRICT { \ .strict = 1, \
@@ -158,7 +163,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function, \ + .error_func = fsck_objects_error_function, \ }#define FSCK_OPTIONS_MISSING_GITMODULES { \ .strict = 1, \
@@ -166,7 +171,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_cb_print_missing_gitmodules, \ + .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ }/* descend in all linked child objects
@@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +__attribute__((format (printf,5 ,6 ))) +int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +const char *checked_ref_name, +enum fsck_msg_id msg_id, +const char *fmt, ...); +/* * Subsystem for storing human-readable names for each object. *
object-file.c +6 -5
@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,* give more context. */ static int hash_format_check_report(struct fsck_options *opts UNUSED, - const struct object_id *oid UNUSED, - enum object_type object_type UNUSED, - enum fsck_msg_type msg_type UNUSED, - enum fsck_msg_id msg_id UNUSED, -const char *message) + const struct object_id *oid UNUSED, + enum object_type object_type UNUSED, + const char *ref_checked_name UNUSED, + enum fsck_msg_type msg_type UNUSED, + enum fsck_msg_id msg_id UNUSED, +const char *message) {error (_ ("object fails fsck: %s" ), message);return 1 ;
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v9 2/9] fsck: add a unified interface for reporting fsck messages
Quoted 38 lines · shejialuo
The static function "report" provided by "
fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "vfsck_report" by adding "checked_ref_name" parameter following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.Like "report", the "
vfsck_report" function will use "error_func" registered in "fsck_options" to report customized messages. Change "error_func" prototype to align with the new "vfsck_report".Then, change "report" function to use "
vfsck_report" to report objects related messages. Add a new function called "fsck_refs_report" to use "vfsck_report" to report refs related messages.Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- builtin/fsck.c | 15 ++++----- builtin/mktag.c | 1 + fsck.c | 81 ++++++++++++++++++++++++++++++++++++------------- fsck.h | 40 +++++++++++++++--------- object-file.c | 11 ++++--- 5 files changed, 101 insertions(+), 47 deletions(-)builtin/fsck.c +0
-3@@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)return -1 ; } -static int fsck_error_func(struct fsck_options *o UNUSED, - const struct object_id *oid, - enum object_type object_type,
builtin/fsck.c static int objerror(struct object *obj, const char *err) · line 98
Quoted 10 lines
builtin/fsck.c +7
-3-enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, - const char *message) +static int fsck_objects_error_func(struct fsck_options *o UNUSED, +const struct object_id *oid, +enum object_type object_type, + const char *checked_ref_name UNUSED, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message)
This is just a suggestion, but I think it would be slightly easier to review if the *_error_func() renames were done in a separate preceding patch. That way the purpose of the renames can also be clearly explained.
Quoted 12 lines
builtin/fsck.c +1
-1{switch (msg_type) {case FSCK_WARN :@@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) fsck_walk_options.walk = mark_object; fsck_obj_options.walk = mark_used; - fsck_obj_options.error_func = fsck_error_func; + fsck_obj_options.error_func = fsck_objects_error_func;if (check_strict) fsck_obj_options.strict =1 ;
[snip]
Quoted 10 lines
builtin/fsck.c +1
-1@@ -166,7 +171,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_cb_print_missing_gitmodules, \ + .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ }/* descend in all linked child objects
builtin/fsck.c int fsck_tag_standalone(const struct object_id *oid, const char *buffer, · line 223
Quoted 10 lines
builtin/fsck.c +7
-0@@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +__attribute__((format (printf,5 ,6 ))) +int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +const char *checked_ref_name, +enum fsck_msg_id msg_id, +const char *fmt, ...); +
I think I mentioned this in a previous reply, but it was missed. Not a big deal, but it might be nice to document int fsck_refs_report() here.
-Justin
Re: [GSoC][PATCH v9 2/9] fsck: add a unified interface for reporting fsck messages
Quoted 46 lines · Justin Tobler
Quoted 45 lines · shejialuo
The static function "report" provided by "
fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "vfsck_report" by adding "checked_ref_name" parameter following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.Like "report", the "
vfsck_report" function will use "error_func" registered in "fsck_options" to report customized messages. Change "error_func" prototype to align with the new "vfsck_report".Then, change "report" function to use "
vfsck_report" to report objects related messages. Add a new function called "fsck_refs_report" to use "vfsck_report" to report refs related messages.Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- builtin/fsck.c | 15 ++++----- builtin/mktag.c | 1 + fsck.c | 81 ++++++++++++++++++++++++++++++++++++------------- fsck.h | 40 +++++++++++++++--------- object-file.c | 11 ++++--- 5 files changed, 101 insertions(+), 47 deletions(-)builtin/fsck.c +4
-6@@ -89,12 +89,13 @@ static int objerror(struct object *obj, const char *err)return -1 ; } -static int fsck_error_func(struct fsck_options *o UNUSED, -const struct object_id *oid, -enum object_type object_type, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +static int fsck_objects_error_func(struct fsck_options *o UNUSED, + const struct object_id *oid, + enum object_type object_type, + const char *checked_ref_name UNUSED,
Quoted 9 lines
builtin/fsck.c static int objerror(struct object *obj, const char *err) · line 98
Quoted 3 lines
builtin/fsck.c +3
-0+enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, + const char *message)This is just a suggestion, but I think it would be slightly easier to review if the
*_error_func()renames were done in a separate preceding patch. That way the purpose of the renames can also be clearly explained.
I agree with this, will change in the next version.
Quoted 26 lines
Quoted 12 lines
builtin/fsck.c +1
-1{switch (msg_type) {case FSCK_WARN :@@ -938,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) fsck_walk_options.walk = mark_object; fsck_obj_options.walk = mark_used; - fsck_obj_options.error_func = fsck_error_func; + fsck_obj_options.error_func = fsck_objects_error_func;if (check_strict) fsck_obj_options.strict =1 ;[snip]
Quoted 13 lines
builtin/fsck.c +1
-1@@ -166,7 +171,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_cb_print_missing_gitmodules, \ + .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ }/* descend in all linked child objects @@ -209,6 +214,13 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options);
Quoted 12 lines
builtin/fsck.c int fsck_tag_standalone(const struct object_id *oid, const char *buffer, · line 223
Quoted 7 lines
builtin/fsck.c +7
-0+__attribute__((format (printf,5 ,6 ))) +int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +const char *checked_ref_name, +enum fsck_msg_id msg_id, +const char *fmt, ...); +I think I mentioned this in a previous reply, but it was missed. Not a big deal, but it might be nice to document
int fsck_refs_report()here.
I will improve this in the next version.
Quoted 1 line
-Justin
Thanks,
Jialuo
PATCH v9 3/9 fsck: add refs-related options and error report function
In reply to shejialuoAdd refs-related options to the "fsck_options", create refs-specific "error_func" callback "fsck_refs_error_function".
"fsck_refs_error_function" will use the "oid" parameter. When the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.
Last, add "FSCK_REFS_OPTIONS_DEFAULT" and "FSCK_REFS_OPTIONS_STRICT" macros to create refs options easily.
--- fsck.c | 23 +++++++++++++++++++++++ fsck.h | 15 +++++++++++++++ 2 files changed, 38 insertions(+)
fsck.c +23 -0
@@ -1252,6 +1252,29 @@ int fsck_objects_error_function(struct fsck_options *o,return 1 ; } +int fsck_refs_error_function(struct fsck_options *options UNUSED, +const struct object_id *oid, + enum object_type object_type UNUSED, +const char *checked_ref_name, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) +{ +static struct strbuf sb =STRBUF_INIT ; + +strbuf_reset (&sb); +strbuf_addstr (&sb, checked_ref_name); +if (oid) +strbuf_addf (&sb," -> (%s)" ,oid_to_hex (oid)); + +if (msg_type ==FSCK_WARN ) { +warning ("%s: %s" , sb.buf, message); +return 0 ; + } +error ("%s: %s" , sb.buf, message); +return 1 ; +} +static int fsck_blobs (struct oidset *blobs_found,struct oidset *blobs_done,enum fsck_msg_id msg_missing,enum fsck_msg_id msg_type,struct fsck_options *options,const char *blob_type)
fsck.h +15 -0
@@ -135,11 +135,19 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); +int fsck_refs_error_function (struct fsck_options *options, +const struct object_id *oid, +enum object_type object_type, +const char *checked_ref_name, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);struct fsck_options {fsck_walk_func walk;fsck_error error_func;unsigned strict:1 ; +unsigned verbose_refs:1 ;enum fsck_msg_type *msg_type;struct oidset skip_oids;struct oidset gitmodules_found;
@@ -173,6 +181,13 @@ struct fsck_options { .gitattributes_done = OIDSET_INIT, \ .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ } +#define FSCK_REFS_OPTIONS_DEFAULT { \ + .error_func = fsck_refs_error_function, \ +} +#define FSCK_REFS_OPTIONS_STRICT { \ + .strict = 1, \ + .error_func = fsck_refs_error_function, \ +}/* descend in all linked child objects * the return value is:
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v9 3/9] fsck: add refs-related options and error report function
Quoted 6 lines · shejialuo
Add refs-related options to the "
fsck_options", create refs-specific "error_func" callback "fsck_refs_error_function"."
fsck_refs_error_function" will use the "oid" parameter. When the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.
Out of curiousity, under what circumstances would the caller want to also pass the oid? Would it simply be to optionally provide additional context?
Quoted 21 lines
Last, add "
FSCK_REFS_OPTIONS_DEFAULT" and "FSCK_REFS_OPTIONS_STRICT" macros to create refs options easily.Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- fsck.c | 23 +++++++++++++++++++++++ fsck.h | 15 +++++++++++++++ 2 files changed, 38 insertions(+)fsck.c +1
-0@@ -1252,6 +1252,29 @@ int fsck_objects_error_function(struct fsck_options *o,return 1 ; } +int fsck_refs_error_function(struct fsck_options *options UNUSED,
fsck.c int fsck_objects_error_function(struct fsck_options *o, · line 1265
Quoted 10 lines
fsck.c +10
-0+const struct object_id *oid, + enum object_type object_type UNUSED, + const char *checked_ref_name, + enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, + const char *message) +{ + static struct strbuf sb = STRBUF_INIT; + +strbuf_reset (&sb);
Naive question, is there reason to reset sb immediately after STRBUF_INIT? My understanding is that because we initialize the buffer, the other fields should also be zeroed. If so, resetting the buffer here seems redundant.
Quoted 26 lines
fsck.c +12
-0+strbuf_addstr (&sb, checked_ref_name); +if (oid) +strbuf_addf (&sb," -> (%s)" ,oid_to_hex (oid)); + +if (msg_type ==FSCK_WARN ) { +warning ("%s: %s" , sb.buf, message); +return 0 ; + } +error ("%s: %s" , sb.buf, message); +return 1 ; +} +static int fsck_blobs (struct oidset *blobs_found,struct oidset *blobs_done,enum fsck_msg_id msg_missing,enum fsck_msg_id msg_type,struct fsck_options *options,const char *blob_type)fsck.h +3
-0@@ -135,11 +135,19 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); +int fsck_refs_error_function(struct fsck_options *options, + const struct object_id *oid, + enum object_type object_type,
fsck.h int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o, · line 150
Quoted 10 lines
fsck.h +5
-0+const char *checked_ref_name, + enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id, + const char *message);struct fsck_options {fsck_walk_func walk;fsck_error error_func;unsigned strict:1 ; +unsigned verbose_refs:1 ;
What is the purpose of adding verbose_refs in this patch? At this point, I'm not seeing it used. If there is a reason to be included in this patch, it might be nice to mention in the commit message.
Quoted 20 lines
fsck.h +7
-0enum fsck_msg_type *msg_type;struct oidset skip_oids;struct oidset gitmodules_found;@@ -173,6 +181,13 @@ struct fsck_options { .gitattributes_done = OIDSET_INIT, \ .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ } +#define FSCK_REFS_OPTIONS_DEFAULT { \ + .error_func = fsck_refs_error_function, \ +} +#define FSCK_REFS_OPTIONS_STRICT { \ + .strict = 1, \ + .error_func = fsck_refs_error_function, \ +}/* descend in all linked child objects * the return value is: Signature · 1 line
2.45.2
Re: [GSoC][PATCH v9 3/9] fsck: add refs-related options and error report function
Quoted 17 lines · Justin Tobler
fsck.c int fsck_objects_error_function(struct fsck_options *o, · line 1265
Quoted 11 lines · shejialuo
fsck.c +11
-0+int fsck_refs_error_function(struct fsck_options *options UNUSED, +const struct object_id *oid, + enum object_type object_type UNUSED, +const char *checked_ref_name, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) +{ +static struct strbuf sb =STRBUF_INIT ; + +strbuf_reset (&sb);Naive question, is there reason to reset
sbimmediately afterSTRBUF_INIT? My understanding is that because we initialize the buffer, the other fields should also be zeroed. If so, resetting the buffer here seems redundant.
This particular strbuf is static, so it needs to be cleared each time the function is called.
The cover letter provides an argument for making it static: that this will be called often, and we don't want to make a lot of repeated allocations. Personally, I find that argument rather weak. Why would an error function be called frequently? Is this really a hot path that needs to worry about a few extra allocations? Also, importantly, every static added makes the code harder to "libify", so making it static requires a very strong reason, but there doesn't seem to be such a reason in this case.
Re: [GSoC][PATCH v9 3/9] fsck: add refs-related options and error report function
Quoted 22 lines · Eric Sunshine
Quoted 17 lines · Justin Tobler
fsck.c int fsck_objects_error_function(struct fsck_options *o, · line 1265
Quoted 11 lines · shejialuo
fsck.c +11
-0+int fsck_refs_error_function(struct fsck_options *options UNUSED, +const struct object_id *oid, + enum object_type object_type UNUSED, +const char *checked_ref_name, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) +{ +static struct strbuf sb =STRBUF_INIT ; + +strbuf_reset (&sb);Naive question, is there reason to reset
sbimmediately afterSTRBUF_INIT? My understanding is that because we initialize the buffer, the other fields should also be zeroed. If so, resetting the buffer here seems redundant.This particular strbuf is static, so it needs to be cleared each time the function is called.
Quoted 8 lines
The cover letter provides an argument for making it static: that this will be called often, and we don't want to make a lot of repeated allocations. Personally, I find that argument rather weak. Why would an error function be called frequently? Is this really a hot path that needs to worry about a few extra allocations? Also, importantly, every static added makes the code harder to "libify", so making it static requires a very strong reason, but there doesn't seem to be such a reason in this case.
I didn't consider the issue of libify. I just want to reduce some memory allocations. I will change this in the next version.
Thanks,
Jialuo
Re: [GSoC][PATCH v9 3/9] fsck: add refs-related options and error report function
In reply to Justin ToblerQuoted 4 lines · Justin Tobler
Quoted 3 lines · shejialuo
Add refs-related options to the "
fsck_options", create refs-specific "error_func" callback "fsck_refs_error_function".
Quoted 8 lines
Quoted 3 lines
"
fsck_refs_error_function" will use the "oid" parameter. When the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.Out of curiousity, under what circumstances would the caller want to also pass the oid? Would it simply be to optionally provide additional context?
Because when we check the refs, we will use "parse_loose_ref_contents" here to check the ref contents. Below is the prototype:
int parse_loose_ref_contents(const char *buf,
struct object_id *oid, ...)
So we could get a oid here. However, we don't know the type of the oid. It may not be commit object but rather a tag object. And I want to provide more flexible operations for caller. When caller passes the oid. The message could be the following:
ref_name -> (oid) : fsck_error_type: user-passed message.
So, actually we have provided additional context for the caller. From another perspective, the object check needs the "oid" parameter, we cannot remove it from the callback "error_func" prototype. So why not just reuse this parameter? It truly provides the caller more flexibility without big changes.
Quoted 10 lines
fsck.h int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o, · line 150
Quoted 5 lines
struct
fsck_options{fsck.h +1
-0fsck_walk_func walk;fsck_error error_func; unsigned strict:1; + unsigned verbose_refs:1;What is the purpose of adding
verbose_refsin this patch? At this point, I'm not seeing it used. If there is a reason to be included in this patch, it might be nice to mention in the commit message.
The reason is that fsck builtin handles the object check but we want to use "git refs verify" command to handle ref check and we put all the real functionality into each file backend. And there is only one entry point in the "git refs verify" command. So we need to use "fsck_options" as the parameter to maintain this state across the ref checks.
Actually, "git-fsck(1)" maintains "verbose" global variable. I think I should not add this option in this patch which may cause a lot of confusion here. I will improve this in the next version.
Quoted 20 lines
Quoted 20 lines
fsck.h +7
-0enum fsck_msg_type *msg_type;struct oidset skip_oids;struct oidset gitmodules_found;@@ -173,6 +181,13 @@ struct fsck_options { .gitattributes_done = OIDSET_INIT, \ .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ } +#define FSCK_REFS_OPTIONS_DEFAULT { \ + .error_func = fsck_refs_error_function, \ +} +#define FSCK_REFS_OPTIONS_STRICT { \ + .strict = 1, \ + .error_func = fsck_refs_error_function, \ +}/* descend in all linked child objects * the return value is: Signature · 1 line
2.45.2
PATCH v9 4/9 refs: set up ref consistency check infrastructure
In reply to shejialuoThe interfaces defined in the ref_storage_be are carefully structured in semantic. It's organized as the five parts:
1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.
To keep consistent with the git-fsck(1), add a new interface named "fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.
--- refs.c | 5 +++++ refs.h | 8 ++++++++ refs/debug.c | 11 +++++++++++ refs/files-backend.c | 15 ++++++++++++++- refs/packed-backend.c | 8 ++++++++ refs/refs-internal.h | 6 ++++++ refs/reftable-backend.c | 8 ++++++++ 7 files changed, 60 insertions(+), 1 deletion(-)
refs.c +5 -0
@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)return check_or_sanitize_refname (refname, flags,NULL ); } +int refs_fsck (struct ref_store *refs,struct fsck_options *o) +{ +return refs->be->fsck (refs, o); +} +void sanitize_refname_component (const char *refname,struct strbuf *out) {if (check_or_sanitize_refname (refname,REFNAME_ALLOW_ONELEVEL , out))
refs.h +8 -0
@@ -4,6 +4,7 @@#include "commit.h" #include "repository.h" +struct fsck_options ;struct object_id ;struct ref_store ;struct strbuf ;
@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat*/ int check_refname_format (const char *refname,int flags); +/* +* Check the reference database for consistency. Return 0 if refs and +* reflogs are consistent, and non-zero otherwise. The errors will be +* written to stderr. +*/ +int refs_fsck (struct ref_store *refs,struct fsck_options *o); +/* * Apply the rules from check_refname_format, but mutate the result until it * is acceptable, and place the result in "out".
refs/debug.c +11 -0
@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,return res; } +static int debug_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store; +int res = drefs->refs->be->fsck (drefs->refs, o); +trace_printf_key (&trace_refs,"fsck: %d\n" , res); +return res; +} +struct ref_storage_be refs_be_debug = { .name ="debug" , .init =NULL ,
@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = { .create_reflog = debug_create_reflog, .delete_reflog = debug_delete_reflog, .reflog_expire = debug_reflog_expire, + + .fsck = debug_fsck, };
refs/files-backend.c +14 -1
@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +static int files_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +int ret; +struct files_ref_store *refs = +files_downcast (ref_store,REF_STORE_READ ,"fsck" ); + + ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); +return ret; +} +struct ref_storage_be refs_be_files = { .name ="files" , .init = files_ref_store_init,
@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = { .reflog_exists = files_reflog_exists, .create_reflog = files_create_reflog, .delete_reflog = files_delete_reflog, - .reflog_expire = files_reflog_expire + .reflog_expire = files_reflog_expire, + + .fsck = files_fsck, };
refs/packed-backend.c +8 -0
@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_sreturn empty_ref_iterator_begin (); } +static int packed_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_packed = { .name ="packed" , .init = packed_ref_store_init,
@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = { .create_reflog = NULL, .delete_reflog = NULL, .reflog_expire = NULL, + + .fsck = packed_fsck, };
refs/refs-internal.h +6 -0
@@ -4,6 +4,7 @@#include "refs.h" #include "iterator.h" +struct fsck_options ;struct ref_transaction ;/*
@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,typedef int read_symbolic_ref_fn (struct ref_store *ref_store,const char *refname,struct strbuf *referent); +typedef int fsck_fn (struct ref_store *ref_store, +struct fsck_options *o); +struct ref_storage_be {const char *name;ref_store_init_fn *init;
@@ -677,6 +681,8 @@ struct ref_storage_be {create_reflog_fn *create_reflog;delete_reflog_fn *delete_reflog;reflog_expire_fn *reflog_expire; + +fsck_fn *fsck; };extern struct ref_storage_be refs_be_files;
refs/reftable-backend.c +8 -0
@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,return ret; } +static int reftable_be_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_reftable = { .name ="reftable" , .init = reftable_be_init,
@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = { .create_reflog = reftable_be_create_reflog, .delete_reflog = reftable_be_delete_reflog, .reflog_expire = reftable_be_reflog_expire, + + .fsck = reftable_be_fsck, };
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v9 4/9] refs: set up ref consistency check infrastructure
Quoted 9 lines · shejialuo
The interfaces defined in the
ref_storage_beare carefully structured in semantic. It's organized as the five parts:1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.
Quoted 4 lines
To keep consistent with the git-fsck(1), add a new interface named "
fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.
In this patch we are not only adding the fsck infrastructure to ref_storage_be, but also wiring what are essentially no-op functions to the various implementations. It might be good to mention this in the commit message so it is better understood that future patches will actually provide proper function implementations.
-Justin
Re: [GSoC][PATCH v9 4/9] refs: set up ref consistency check infrastructure
Quoted 15 lines · Justin Tobler
Quoted 9 lines · shejialuo
The interfaces defined in the
ref_storage_beare carefully structured in semantic. It's organized as the five parts:1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.Quoted 4 lines
To keep consistent with the git-fsck(1), add a new interface named "
fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.
Quoted 6 lines
In this patch we are not only adding the
fsckinfrastructure toref_storage_be, but also wiring what are essentially no-op functions to the various implementations. It might be good to mention this in the commit message so it is better understood that future patches will actually provide proper function implementations.
Yes, I agree. I will improve this in the next version.
Quoted 1 line
-Justin
PATCH v9 5/9 builtin/refs: add verify subcommand
In reply to shejialuoIntroduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency.
--- Documentation/git-refs.txt | 13 +++++++++++ builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 57 insertions(+)
Documentation/git-refs.txt +13 -0
@@ -10,6 +10,7 @@ SYNOPSIS -------- [verse] 'git refs migrate' --ref-format=<format> [--dry-run] +'git refs verify' [--strict] [--verbose] DESCRIPTION -----------
@@ -22,6 +23,9 @@ COMMANDS migrate:: Migrate ref store between different formats. +verify:: + Verify reference database consistency. + OPTIONS -------
@@ -39,6 +43,15 @@ include::ref-storage-format.txt[] can be used to double check that the migration works as expected before performing the actual migration. +The following options are specific to 'git refs verify': + +--strict:: + Enable more strict checking, every WARN severity for the `Fsck Messages` + be seen as ERROR. See linkgit:git-fsck[1]. + +--verbose:: + When verifying the reference database consistency, be chatty. + KNOWN LIMITATIONS -----------------
builtin/refs.c +44 -0
69 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v9 5/9] builtin/refs: add verify subcommand
Quoted 2 lines · shejialuo
Introduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency.
The next patch in the series uses git-refs verify to when running git-fsck(1). It might be worth mentioning here that this is also intended as the entry point for fscking refs.
Quoted 4 lines
Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>
[snip]
Quoted 8 lines
builtin/refs.c +4
-0@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)return err; } +static int cmd_refs_verify (int argc,const char **argv,const char *prefix) +{ +struct fsck_options fsck_refs_options =FSCK_REFS_OPTIONS_DEFAULT ; +const char *const verify_usage[] = {
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 79
Quoted 10 lines
builtin/refs.c +10
-0+REFS_VERIFY_USAGE , + NULL, + }; +unsigned int verbose =0 , strict =0 ; +struct option options[] = { +OPT__VERBOSE (&verbose,N_ ("be verbose" )), +OPT_BOOL (0 ,"strict" , &strict,N_ ("enable strict checking" )), +OPT_END (), + }; +int ret =0 ;
nit: Being that we always assign a value to ret, defaulting here to zero is redundant.
Quoted 7 lines
builtin/refs.c +7
-0+ + argc =parse_options (argc, argv, prefix, options, verify_usage,0 ); +if (argc) +usage (_ ("too many arguments" )); + +if (verbose) + fsck_refs_options.verbose_refs =1 ;
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 96
Quoted 10 lines
builtin/refs.c +10
-0+if (strict) + fsck_refs_options.strict =1 ; + +git_config (git_fsck_config, &fsck_refs_options); +prepare_repo_settings (the_repository); + + ret =refs_fsck (get_main_ref_store (the_repository), &fsck_refs_options); + +/* +* Explicitly free the allocated array and "oid_skiplist"
s/oid_skiplist/skip_oids
I think we forgot to update this comment after the variable was renamed in a previous patch version.
Quoted 23 lines
builtin/refs.c +8
-0+*/ +free (fsck_refs_options.msg_type); +oidset_clear (&fsck_refs_options.skip_oids); +return ret; +} +int cmd_refs (int argc,const char **argv,const char *prefix) {const char *const refs_usage[] = {REFS_MIGRATE_USAGE , +REFS_VERIFY_USAGE ,NULL , };parse_opt_subcommand_fn *fn =NULL ;struct option opts[] = {OPT_SUBCOMMAND ("migrate" , &fn, cmd_refs_migrate), +OPT_SUBCOMMAND ("verify" , &fn, cmd_refs_verify),OPT_END (), };Signature · 1 line
2.45.2
Re: [GSoC][PATCH v9 5/9] builtin/refs: add verify subcommand
Quoted 8 lines · Justin Tobler
Quoted 2 lines · shejialuo
Introduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency.
The next patch in the series uses
git-refs verifyto when running git-fsck(1). It might be worth mentioning here that this is also intended as the entry point for fscking refs.
Yes, I will improve the commit message, in the next patch.
Quoted 18 lines
Quoted 4 lines
Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>[snip]
Quoted 13 lines
builtin/refs.c +9
-0@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)return err; } +static int cmd_refs_verify (int argc,const char **argv,const char *prefix) +{ +struct fsck_options fsck_refs_options =FSCK_REFS_OPTIONS_DEFAULT ; +const char *const verify_usage[] = { +REFS_VERIFY_USAGE , +NULL , + }; +unsigned int verbose =0 , strict =0 ; +struct option options[] = {
Quoted 9 lines
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 79
Quoted 5 lines
builtin/refs.c +5
-0+OPT__VERBOSE (&verbose,N_ ("be verbose" )), +OPT_BOOL (0 ,"strict" , &strict,N_ ("enable strict checking" )), + OPT_END(), + }; +int ret =0 ;nit: Being that we always assign a value to
ret, defaulting here to zero is redundant.
I agree.
Quoted 20 lines
Quoted 7 lines
builtin/refs.c +7
-0+ + argc =parse_options (argc, argv, prefix, options, verify_usage,0 ); +if (argc) +usage (_ ("too many arguments" )); + +if (verbose) + fsck_refs_options.verbose_refs =1 ;builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 96
Quoted 10 lines
builtin/refs.c +10
-0+if (strict) + fsck_refs_options.strict =1 ; + +git_config (git_fsck_config, &fsck_refs_options); +prepare_repo_settings (the_repository); + + ret =refs_fsck (get_main_ref_store (the_repository), &fsck_refs_options); + +/* +* Explicitly free the allocated array and "oid_skiplist" s/oid_skiplist/skip_oids
Quoted 3 lines
I think we forgot to update this comment after the variable was renamed in a previous patch version.
Yes, thanks for you remind. I have just forgotten here.
PATCH v9 6/9 builtin/fsck: add `git-refs verify` child process
In reply to shejialuoIntroduce a new function "fsck_refs" that initializes and runs a child process to execute the "git-refs verify" command.
--- builtin/fsck.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
builtin/fsck.c +17 -0
@@ -897,6 +897,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)return res; } +static void fsck_refs (void ) +{ +struct child_process refs_verify =CHILD_PROCESS_INIT ; +child_process_init (&refs_verify); + refs_verify.git_cmd =1 ; +strvec_pushl (&refs_verify.args,"refs" ,"verify" ,NULL ); +if (verbose) +strvec_push (&refs_verify.args,"--verbose" ); +if (check_strict) +strvec_push (&refs_verify.args,"--strict" ); + +if (run_command (&refs_verify)) + errors_found |=ERROR_REFS ; +} +static char const *const fsck_usage[] = { N_("git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\n" " [--[no-]full] [--strict] [--verbose] [--lost-found]\n"
@@ -1066,6 +1081,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)check_connectivity (); +fsck_refs (); +if (the_repository->settings.core_commit_graph) {struct child_process commit_graph_verify =CHILD_PROCESS_INIT ;
Signature · 1 line
2.45.2
PATCH v9 7/9 files-backend: add unified interface for refs scanning
In reply to shejialuoFor refs and reflogs, we need to scan its corresponding directories to check every regular file or symbolic link which shares the same pattern. Introduce a unified interface for scanning directories for files-backend.
--- refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 76 insertions(+), 1 deletion(-)
refs/files-backend.c +76 -1
@@ -6,6 +6,7 @@#include "../gettext.h" #include "../hash.h" #include "../hex.h" +#include "../fsck.h" #include "../refs.h" #include "refs-internal.h" #include "ref-cache.h"
@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +/* +* For refs and reflogs, they share a unified interface when scanning +* the whole directory. This function is used as the callback for each +* regular file or symlink in the directory. +*/ +typedef int (*files_fsck_refs_fn )(struct fsck_options *o, +const char *gitdir, +const char *refs_check_dir, +struct dir_iterator *iter); + +static int files_fsck_refs_dir (struct ref_store *ref_store, +struct fsck_options *o, +const char *refs_check_dir, +files_fsck_refs_fn *fsck_refs_fns) +{ +const char *gitdir = ref_store->gitdir; +struct strbuf sb =STRBUF_INIT ; +struct dir_iterator *iter; +int iter_status; +int ret =0 ; + +strbuf_addf (&sb,"%s/%s" , gitdir, refs_check_dir); + + iter =dir_iterator_begin (sb.buf,0 ); + +if (!iter) { + ret =error_errno ("cannot open directory %s" , sb.buf); + goto out; + } + +while ((iter_status =dir_iterator_advance (iter)) ==ITER_OK ) { +if (S_ISDIR (iter->st.st_mode)) { +continue ; + }else if (S_ISREG (iter->st.st_mode) || +S_ISLNK (iter->st.st_mode)) { +if (o->verbose_refs) +fprintf_ln (stderr,"Checking %s/%s" , + refs_check_dir, iter->relative_path); +for (size_t i =0 ; fsck_refs_fns[i]; i++) { +if (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter)) + ret =-1 ; + } + }else { + ret =error (_ ("unexpected file type for '%s'" ), + iter->basename); + } + } + +if (iter_status !=ITER_DONE ) + ret =error (_ ("failed to iterate over '%s'" ), sb.buf); + +out: +strbuf_release (&sb); +return ret; +} + +static int files_fsck_refs (struct ref_store *ref_store, +struct fsck_options *o) +{ +int ret; +files_fsck_refs_fn fsck_refs_fns[]= { +NULL + }; + +if (o->verbose_refs) +fprintf_ln (stderr,"Checking references consistency" ); + + ret =files_fsck_refs_dir (ref_store, o,"refs" , fsck_refs_fns); + +return ret; +} +static int files_fsck (struct ref_store *ref_store,struct fsck_options *o) {
@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,struct files_ref_store *refs =files_downcast (ref_store,REF_STORE_READ ,"fsck" ); - ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); + ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o) + |files_fsck_refs (ref_store, o); +return ret; }
Signature · 1 line
2.45.2
PATCH v9 8/9 fsck: add ref name check for files backend
In reply to shejialuoThe git-fsck(1) only implicitly checks the reference, it does not fully check refs with bad format name such as standalone "@" and name ending with ".lock".
In order to provide such checks, add a new fsck message id "badRefName" with default ERROR type. Use existing "check_refname_format" to explicit check the ref name. And add a new unit test to verify the functionality.
--- Documentation/fsck-msgids.txt | 3 + fsck.h | 1 + refs/files-backend.c | 20 +++++++ t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++ 4 files changed, 125 insertions(+) create mode 100755 t/t0602-reffiles-fsck.sh
Documentation/fsck-msgids.txt +3 -0
@@ -19,6 +19,9 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefName`:: + (ERROR) A ref has a bad name. + `badTagName`:: (INFO) A tag has an invalid format.
fsck.h +1 -0
@@ -31,6 +31,7 @@ enum fsck_msg_type { FUNC(BAD_NAME, ERROR) \ FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ + FUNC(BAD_REF_NAME, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
refs/files-backend.c +20 -0
@@ -3419,6 +3419,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,const char *refs_check_dir, struct dir_iterator *iter); +static int files_fsck_refs_name (struct fsck_options *o, + const char *gitdir UNUSED, +const char *refs_check_dir, +struct dir_iterator *iter) +{ +struct strbuf sb =STRBUF_INIT ; +int ret =0 ; + +if (check_refname_format (iter->basename,REFNAME_ALLOW_ONELEVEL )) { +strbuf_addf (&sb,"%s/%s" , refs_check_dir, iter->relative_path); + ret =fsck_refs_report (o,NULL , sb.buf, +FSCK_MSG_BAD_REF_NAME , +"invalid refname format" ); + } + +strbuf_release (&sb); +return ret; +} + static int files_fsck_refs_dir(struct ref_store *ref_store, struct fsck_options *o, const char *refs_check_dir,
@@ -3470,6 +3489,7 @@ static int files_fsck_refs(struct ref_store *ref_store, {int ret;files_fsck_refs_fn fsck_refs_fns[]= { + files_fsck_refs_name,NULL };
t/t0602-reffiles-fsck.sh new file +101 -0
101 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v9 9/9 fsck: add ref content check for files backend
In reply to shejialuoEnhance the git-fsck(1) command by adding a check for reference content in the files backend. The new functionality ensures that symrefs, real symbolic link and regular refs are validated correctly.
In order to check the trailing content of the regular refs, add a new parameter trailing to parse_loose_ref_contents.
For symrefs, parse_loose_ref_contents will set the "referent". However, symbolic link could be either absolute or relative. Use "strbuf_add_real_path" to read the symbolic link and convert the relative path to absolute path. Then use "skip_prefix" to make it align with symref "referent".
Thus, the symrefs and symbolic links could share the same interface. Add a new function "files_fsck_symref_target" which aims at checking the following things:
1. whether the pointee is under the refs/ directory.
2. whether the pointee name is correct.
3. whether the pointee path is a wrong type in filesystem.
Last, add the following FSCK MESSAGEs:
1. "badRefContent(ERROR)": A ref has a bad content
2. "badSymrefPointee(ERROR)": The pointee of a symref is bad.
3. "trailingRefContent(WARN)": A ref content has trailing contents.
--- Documentation/fsck-msgids.txt | 9 +++ fsck.h | 3 + refs.c | 2 +- refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++- refs/refs-internal.h | 5 +- t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++ 6 files changed, 269 insertions(+), 5 deletions(-)
Documentation/fsck-msgids.txt +9 -0
@@ -19,9 +19,15 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefContent`:: + (ERROR) A ref has a bad content. + `badRefName`:: (ERROR) A ref has a bad name. +`badSymrefPointee`:: + (ERROR) The pointee of a symref is bad. + `badTagName`:: (INFO) A tag has an invalid format.
@@ -167,6 +173,9 @@ `nullSha1`:: (WARN) Tree contains entries pointing to a null sha1. +`trailingRefContent`:: + (WARN) A ref content has trailing contents. + `treeNotSorted`:: (ERROR) A tree is not properly sorted.
fsck.h +3 -0
@@ -32,6 +32,8 @@ enum fsck_msg_type { FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ FUNC(BAD_REF_NAME, ERROR) \ + FUNC(BAD_REF_CONTENT, ERROR) \ + FUNC(BAD_SYMREF_POINTEE, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
@@ -72,6 +74,7 @@ enum fsck_msg_type { FUNC(HAS_DOTDOT, WARN) \ FUNC(HAS_DOTGIT, WARN) \ FUNC(NULL_SHA1, WARN) \ + FUNC(TRAILING_REF_CONTENT, WARN) \ FUNC(ZERO_PADDED_FILEMODE, WARN) \ FUNC(NUL_IN_COMMIT, WARN) \FUNC (LARGE_PATHNAME ,WARN ) \
refs.c +1 -1
@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store, } result =parse_loose_ref_contents (content.buf, oid, referent, type, - failure_errno); + failure_errno,NULL ); done:strbuf_release (&full_path);
refs/files-backend.c +143 -2
181 lines not on this page · raw message
refs/refs-internal.h +3 -2
@@ -709,11 +709,12 @@ struct ref_store {/* * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for -* invalid contents. +* invalid contents. Also *trailing is set to the first character after the +* refname or NULL if the referent is not empty. */ int parse_loose_ref_contents (const char *buf,struct object_id *oid,struct strbuf *referent,unsigned int *type, -int *failure_errno); +int *failure_errno,const char **trailing);/* * Fill in the generic part of refs and add it to our collection of
t/t0602-reffiles-fsck.sh +110 -0
114 lines not on this page · raw message
Signature · 1 line
2.45.2
v10 cover ref consistency check infra setup
In reply to shejialuoHi All:
This version handles the following problems:
1. Following the Justin's advice. Add a new commit to rename the objects-related fsck error functions to make the series more clean.
2. Add a simple documentation for "fsck_refs_report".
3. The previous implementation of "fsck_refs_error_function" used "static struct strbuf", as Eric said, it's a bad idea to use "static" because of infrequent usage of "fsck_refs_error_function". And it will make the code arder to "libify". I didn't consider this. So In this version, use "struct strbuf" instead and add the corresponding memory-free operation.
4. I should not add "verbose_refs" option in "fsck: add refs-related error report function" commit. I add this option in "builtin/refs" to make no confusion.
5. Enchance the commit message for "refs: set up ref consistency check infrastructure" to mention that we have added placeholder functions.
6. Change the "oid_skiplist" from comments which are ignored by the previous version.
Thanks for every reviwer.
CI: https://github.com/shejialuo/git/pull/8
Thanks,
Jialuo
shejialuo (10):
fsck: rename "skiplist" to "skip_oids"
fsck: rename objects-related fsck error functions
fsck: add a unified interface for reporting fsck messages
fsck: add refs-related error report function
refs: set up ref consistency check infrastructure builtin/refs: add verify subcommand and verbose_refs for
"fsck_options"
builtin/fsck: add git-refs verify child process
files-backend: add unified interface for refs scanning
fsck: add ref name check for files backend
fsck: add ref content check for files backend
Documentation/fsck-msgids.txt | 12 ++ Documentation/git-refs.txt | 13 ++ builtin/fsck.c | 32 ++++- builtin/mktag.c | 1 + builtin/refs.c | 44 ++++++ fsck.c | 109 ++++++++++++--- fsck.h | 66 ++++++--- object-file.c | 11 +- refs.c | 7 +- refs.h | 8 ++ refs/debug.c | 11 ++ refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++- refs/packed-backend.c | 8 ++ refs/refs-internal.h | 11 +- refs/reftable-backend.c | 8 ++ t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++ 16 files changed, 750 insertions(+), 57 deletions(-) create mode 100755 t/t0602-reffiles-fsck.sh
Range-diff against v9:
1: e044f933de = 1: e044f933de fsck: rename "skiplist" to "skip_oids" -: ---------- > 2: 73a7c53a23 fsck: rename objects-related fsck error functions 2: daaf3d0ffe ! 3: df4837e960 fsck: add a unified interface for reporting fsck messages
@@ Commit message
## builtin/fsck.c ##
@@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)
- return -1;
- }
-
--static int fsck_error_func(struct fsck_options *o UNUSED,
-- const struct object_id *oid,
-- enum object_type object_type,
-- enum fsck_msg_type msg_type,
-- enum fsck_msg_id msg_id UNUSED,
-- const char *message)
-+static int fsck_objects_error_func(struct fsck_options *o UNUSED,
-+ const struct object_id *oid,
-+ enum object_type object_type,
+ static int fsck_objects_error_func(struct fsck_options *o UNUSED,
+ const struct object_id *oid,
+ enum object_type object_type,
+ const char *checked_ref_name UNUSED,
-+ enum fsck_msg_type msg_type,
-+ enum fsck_msg_id msg_id UNUSED,
-+ const char *message)
- {
- switch (msg_type) {
- case FSCK_WARN:
-@@ builtin/fsck.c: int cmd_fsck(int argc, const char **argv, const char *prefix) -
- fsck_walk_options.walk = mark_object;
- fsck_obj_options.walk = mark_used;
-- fsck_obj_options.error_func = fsck_error_func;
-+ fsck_obj_options.error_func = fsck_objects_error_func;
- if (check_strict)
- fsck_obj_options.strict = 1;
-
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
## builtin/mktag.c ##
@@ builtin/mktag.c: static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;
@@ fsck.c: static int report(struct fsck_options *options,
{
if (!options->object_names)
@@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,
- type);
- }
-
--int fsck_error_function(struct fsck_options *o,
-- const struct object_id *oid,
-- enum object_type object_type UNUSED,
-- enum fsck_msg_type msg_type,
-- enum fsck_msg_id msg_id UNUSED,
-- const char *message)
-+int fsck_objects_error_function(struct fsck_options *o,
-+ const struct object_id *oid,
-+ enum object_type object_type UNUSED,
+ int fsck_objects_error_function(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type UNUSED,
+ const char *checked_ref_name UNUSED,
-+ enum fsck_msg_type msg_type,
-+ enum fsck_msg_id msg_id UNUSED,
-+ const char *message)
- {
- if (msg_type == FSCK_WARN) {
- warning("object %s: %s", fsck_describe_object(o, oid), message);
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
@@ fsck.c: int git_fsck_config(const char *var, const char *value,
- * Custom error callbacks that are used in more than one place.
- */
-
--int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
-- const struct object_id *oid,
-- enum object_type object_type,
-- enum fsck_msg_type msg_type,
-- enum fsck_msg_id msg_id,
-- const char *message)
-+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
-+ const struct object_id *oid,
-+ enum object_type object_type,
+ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
+ const char *checked_ref_name,
-+ enum fsck_msg_type msg_type,
-+ enum fsck_msg_id msg_id,
-+ const char *message)
- {
- if (msg_id == FSCK_MSG_GITMODULES_MISSING) {
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message)
+@@ fsck.c: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
puts(oid_to_hex(oid));
return 0;
}
-- return fsck_error_function(o, oid, object_type, msg_type, msg_id, message);
+- return fsck_objects_error_function(o, oid, object_type,
+ return fsck_objects_error_function(o, oid, object_type, checked_ref_name,
-+ msg_type, msg_id, message);
+ msg_type, msg_id, message);
}
## fsck.h ##
@@ fsck.h: int is_valid_msg_type(const char *msg_id, const char *msg_type);
enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message);
--int fsck_error_function(struct fsck_options *o,
-- const struct object_id *oid, enum object_type object_type,
-- enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
-- const char *message);
--int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
-- const struct object_id *oid,
-- enum object_type object_type,
-- enum fsck_msg_type msg_type,
-- enum fsck_msg_id msg_id,
-- const char *message);
-+int fsck_objects_error_function(struct fsck_options *o,
-+ const struct object_id *oid, enum object_type object_type,
+ int fsck_objects_error_function(struct fsck_options *o,
+ const struct object_id *oid, enum object_type object_type,
+ const char *checked_ref_name,
-+ enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
-+ const char *message);
-+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
-+ const struct object_id *oid,
-+ enum object_type object_type,
+ enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
+ const char *message);
+ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
+ const char *checked_ref_name,
-+ enum fsck_msg_type msg_type,
-+ enum fsck_msg_id msg_id,
-+ const char *message);
-
- struct fsck_options {
- fsck_walk_func walk;
-@@ fsck.h: struct fsck_options {
- .gitmodules_done = OIDSET_INIT, \
- .gitattributes_found = OIDSET_INIT, \
- .gitattributes_done = OIDSET_INIT, \
-- .error_func = fsck_error_function \
-+ .error_func = fsck_objects_error_function \
- }
- #define FSCK_OPTIONS_STRICT { \
- .strict = 1, \
-@@ fsck.h: struct fsck_options {
- .gitmodules_done = OIDSET_INIT, \
- .gitattributes_found = OIDSET_INIT, \
- .gitattributes_done = OIDSET_INIT, \
-- .error_func = fsck_error_function, \
-+ .error_func = fsck_objects_error_function, \
- }
- #define FSCK_OPTIONS_MISSING_GITMODULES { \
- .strict = 1, \
-@@ fsck.h: struct fsck_options {
- .gitmodules_done = OIDSET_INIT, \
- .gitattributes_found = OIDSET_INIT, \
- .gitattributes_done = OIDSET_INIT, \
-- .error_func = fsck_error_cb_print_missing_gitmodules, \
-+ .error_func = fsck_objects_error_cb_print_missing_gitmodules, \
- }
-
- /* descend in all linked child objects
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message);
@@ fsck.h: int fsck_tag_standalone(const struct object_id *oid, const char *buffer,
*/
int fsck_finish(struct fsck_options *options);
++/*
++ * Report an error or warning for refs.
++ */
+__attribute__((format (printf, 5, 6)))
+int fsck_refs_report(struct fsck_options *options,
+ const struct object_id *oid,
3: 40da85ae30 ! 4: c8eb01c987 fsck: add refs-related options and error report function
@@ Metadata
Author: shejialuo <[email protected]>
## Commit message ##
- fsck: add refs-related options and error report function
+ fsck: add refs-related error report function
Add refs-related options to the "fsck_options", create refs-specific
"error_func" callback "fsck_refs_error_function".
@@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
+{
-+ static struct strbuf sb = STRBUF_INIT;
++ struct strbuf sb = STRBUF_INIT;
++ int ret = 0;
+
-+ strbuf_reset(&sb);
+ strbuf_addstr(&sb, checked_ref_name);
+ if (oid)
+ strbuf_addf(&sb, " -> (%s)", oid_to_hex(oid));
+
-+ if (msg_type == FSCK_WARN) {
++ if (msg_type == FSCK_WARN)
+ warning("%s: %s", sb.buf, message);
-+ return 0;
-+ }
-+ error("%s: %s", sb.buf, message);
-+ return 1;
++ else
++ ret = error("%s: %s", sb.buf, message);
++
++ strbuf_release(&sb);
++ return ret;
+}
+
static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done,
@@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *
struct fsck_options {
fsck_walk_func walk;
- fsck_error error_func;
- unsigned strict:1;
-+ unsigned verbose_refs:1;
- enum fsck_msg_type *msg_type;
- struct oidset skip_oids;
- struct oidset gitmodules_found;
@@ fsck.h: struct fsck_options {
.gitattributes_done = OIDSET_INIT, \
.error_func = fsck_objects_error_cb_print_missing_gitmodules, \ 4: a38ea1b117 ! 5: e4085df496 refs: set up ref consistency check infrastructure
@@ Commit message
grouped into any above five categories. Explicitly add blank line to make it different from others.
+ Last, implement placeholder functions for each ref backends.
+
5: 8320f56e0b ! 6: 497f224bed builtin/refs: add verify subcommand
@@ Metadata
Author: shejialuo <[email protected]>
## Commit message ##
- builtin/refs: add verify subcommand
+ builtin/refs: add verify subcommand and verbose_refs for "fsck_options"
Introduce a new subcommand "verify" in git-refs(1) to allow the user to
- check the reference database consistency.
+ check the reference database consistency and also this subcommand will
+ be used as the entry point of checking refs for "git-fsck(1)". Last, add
+ "verbose_refs" field into "fsck_options" to indicate whether we should
+ print verbose messages when checking refs consistency.
@@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c
+ OPT_BOOL(0, "strict", &strict, N_("enable strict checking")),
+ OPT_END(),
+ };
-+ int ret = 0;
++ int ret;
+
+ argc = parse_options(argc, argv, prefix, options, verify_usage, 0);
+ if (argc)
@@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c
+ ret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);
+
+ /*
-+ * Explicitly free the allocated array and "oid_skiplist"
++ * Explicitly free the allocated array and "skip_oids" set
+ */
+ free(fsck_refs_options.msg_type);
+ oidset_clear(&fsck_refs_options.skip_oids);
@@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c
OPT_END(),
};
+
+ ## fsck.h ##
+@@ fsck.h: struct fsck_options {
+ fsck_walk_func walk;
+ fsck_error error_func;
+ unsigned strict:1;
++ unsigned verbose_refs:1;
+ enum fsck_msg_type *msg_type;
+ struct oidset skip_oids;
+ struct oidset gitmodules_found;
6: 6614a06ef5 = 7: 86a14c7b43 builtin/fsck: add `git-refs verify` child process 7: 928cc96396 = 8: daedb80b47 files-backend: add unified interface for refs scanning 8: 4d50d4932f = 9: c36d588e4f fsck: add ref name check for files backend 9: 7edb810819 = 10: 521e0d9ca3 fsck: add ref content check for files backend
Signature · 1 line
2.45.2
PATCH v10 1/10 fsck: rename "skiplist" to "skip_oids"
The "skiplist" field in "fsck_options" is related to objects. Because we are going to introduce ref consistency check, the "skiplist" name is too general which will make the caller think "skiplist" is related to both the refs and objects.
It may seem that for both refs and objects, we should provide a general "skiplist" here. However, the type for "skiplist" is struct oidset which is totally unsuitable for refs.
To avoid above ambiguity, rename "skiplist" to "skip_oids".
--- fsck.c | 4 ++-- fsck.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
fsck.c +2 -2
@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)if (!strcmp (buf,"skiplist" )) {if (equal == len)die ("skiplist requires a path" ); -oidset_parse_file (&options->skiplist, buf + equal +1 , +oidset_parse_file (&options->skip_oids, buf + equal +1 , the_repository->hash_algo); buf += len +1 ;continue ;
@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)static int object_on_skiplist (struct fsck_options *opts,const struct object_id *oid) { -return opts && oid &&oidset_contains (&opts->skiplist, oid); +return opts && oid &&oidset_contains (&opts->skip_oids, oid); } __attribute__((format (printf, 5, 6)))
fsck.h +2 -2
@@ -136,7 +136,7 @@ struct fsck_options {fsck_error error_func; unsigned strict:1;enum fsck_msg_type *msg_type; -struct oidset skiplist; +struct oidset skip_oids;struct oidset gitmodules_found;struct oidset gitmodules_done;struct oidset gitattributes_found;
@@ -145,7 +145,7 @@ struct fsck_options { };#define FSCK_OPTIONS_DEFAULT { \ - .skiplist = OIDSET_INIT, \ + .skip_oids = OIDSET_INIT, \ .gitmodules_found = OIDSET_INIT, \ .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \
Signature · 1 line
2.45.2
PATCH v10 2/10 fsck: rename objects-related fsck error functions
In reply to shejialuoThe names of objects-related fsck error functions are general. It's OK when there is only object database check. However, we are going to introduce refs database check. In order to avoid ambiguity, rename objects-related fsck error functions to explicitly indicate these functions are used to report objects-related messages.
--- builtin/fsck.c | 14 +++++++------- fsck.c | 27 ++++++++++++++------------- fsck.h | 26 +++++++++++++------------- 3 files changed, 34 insertions(+), 33 deletions(-)
builtin/fsck.c +7 -7
@@ -89,12 +89,12 @@ static int objerror(struct object *obj, const char *err)return -1 ; } -static int fsck_error_func(struct fsck_options *o UNUSED, -const struct object_id *oid, -enum object_type object_type, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +static int fsck_objects_error_func(struct fsck_options *o UNUSED, +const struct object_id *oid, +enum object_type object_type, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {switch (msg_type) {case FSCK_WARN :
@@ -938,7 +938,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) fsck_walk_options.walk = mark_object; fsck_obj_options.walk = mark_used; - fsck_obj_options.error_func = fsck_error_func; + fsck_obj_options.error_func = fsck_objects_error_func;if (check_strict) fsck_obj_options.strict =1 ;
fsck.c +14 -13
@@ -1200,12 +1200,12 @@ int fsck_buffer(const struct object_id *oid, enum object_type type, type); } -int fsck_error_function (struct fsck_options *o, -const struct object_id *oid, - enum object_type object_type UNUSED, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +int fsck_objects_error_function (struct fsck_options *o, +const struct object_id *oid, + enum object_type object_type UNUSED, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {if (msg_type ==FSCK_WARN ) {warning ("object %s: %s" ,fsck_describe_object (o, oid), message);
@@ -1303,16 +1303,17 @@ int git_fsck_config(const char *var, const char *value,* Custom error callbacks that are used in more than one place. */ -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message) +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +const struct object_id *oid, +enum object_type object_type, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message) {if (msg_id ==FSCK_MSG_GITMODULES_MISSING ) {puts (oid_to_hex (oid));return 0 ; } -return fsck_error_function (o, oid, object_type, msg_type, msg_id, message); +return fsck_objects_error_function (o, oid, object_type, + msg_type, msg_id, message); }
fsck.h +13 -13
@@ -120,16 +120,16 @@ typedef int (*fsck_error)(struct fsck_options *o, enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); -int fsck_error_function (struct fsck_options *o, -const struct object_id *oid,enum object_type object_type, -enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, -const char *message); -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message); +int fsck_objects_error_function (struct fsck_options *o, +const struct object_id *oid,enum object_type object_type, +enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, +const char *message); +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +const struct object_id *oid, +enum object_type object_type, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);struct fsck_options {fsck_walk_func walk;
@@ -150,7 +150,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function \ + .error_func = fsck_objects_error_function \ }#define FSCK_OPTIONS_STRICT { \ .strict = 1, \
@@ -158,7 +158,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function, \ + .error_func = fsck_objects_error_function, \ }#define FSCK_OPTIONS_MISSING_GITMODULES { \ .strict = 1, \
@@ -166,7 +166,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_cb_print_missing_gitmodules, \ + .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ }/* descend in all linked child objects
Signature · 1 line
2.45.2
PATCH v10 3/10 fsck: add a unified interface for reporting fsck messages
In reply to shejialuoThe static function "report" provided by "fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "vfsck_report" by adding "checked_ref_name" parameter following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.
Like "report", the "vfsck_report" function will use "error_func" registered in "fsck_options" to report customized messages. Change "error_func" prototype to align with the new "vfsck_report".
Then, change "report" function to use "vfsck_report" to report objects related messages. Add a new function called "fsck_refs_report" to use "vfsck_report" to report refs related messages.
--- builtin/fsck.c | 1 + builtin/mktag.c | 1 + fsck.c | 56 +++++++++++++++++++++++++++++++++++++++++-------- fsck.h | 17 ++++++++++++++- object-file.c | 11 +++++----- 5 files changed, 71 insertions(+), 15 deletions(-)
builtin/fsck.c +1 -0
@@ -92,6 +92,7 @@ static int objerror(struct object *obj, const char *err) static int fsck_objects_error_func(struct fsck_options *o UNUSED,const struct object_id *oid,enum object_type object_type, + const char *checked_ref_name UNUSED,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message)
builtin/mktag.c +1 -0
@@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT; static int mktag_fsck_error_func(struct fsck_options *o UNUSED, const struct object_id *oid UNUSED, enum object_type object_type UNUSED, + const char *checked_ref_name UNUSED,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message)
fsck.c +47 -9
92 lines not on this page · raw message
fsck.h +16 -1
@@ -114,19 +114,24 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);typedef int (*fsck_walk_func )(struct object *obj,enum object_type object_type,void *data,struct fsck_options *options); -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */ +/* +* callback function for reporting errors when checking either objects or refs +*/ typedef int (*fsck_error )(struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const char *checked_ref_name,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);int fsck_objects_error_function (struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const char *checked_ref_name,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const char *checked_ref_name,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);
@@ -209,6 +214,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +/* +* Report an error or warning for refs. +*/ +__attribute__((format (printf,5 ,6 ))) +int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +const char *checked_ref_name, +enum fsck_msg_id msg_id, +const char *fmt, ...); +/* * Subsystem for storing human-readable names for each object. *
object-file.c +6 -5
@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,* give more context. */ static int hash_format_check_report(struct fsck_options *opts UNUSED, - const struct object_id *oid UNUSED, - enum object_type object_type UNUSED, - enum fsck_msg_type msg_type UNUSED, - enum fsck_msg_id msg_id UNUSED, -const char *message) + const struct object_id *oid UNUSED, + enum object_type object_type UNUSED, + const char *ref_checked_name UNUSED, + enum fsck_msg_type msg_type UNUSED, + enum fsck_msg_id msg_id UNUSED, +const char *message) {error (_ ("object fails fsck: %s" ), message);return 1 ;
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v10 03/10] fsck: add a unified interface for reporting fsck messages
Quoted 7 lines · shejialuo
The static function "report" provided by "
fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "vfsck_report" by adding "checked_ref_name" parameter following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.
Like strbuf_vinsertf(), it is a good idea to have "v" in the name of a function that takes va_list, but fsck_vreport() would probably be a better name here. Arguably, the original report() is misnamed (as a printf-like function that takes format string, it probably would have wanted to be reportf() instead), but unless we are fixing that at the same time, calling this fsck_vreportf() would probably be too much. Consistently misnaming it by omitting the final "f" would be fine.
At this step it is still not clear if the previous step was really needed; you have this "v" thing that is designed to be usable by both reporting issues around objects and issues around refs, but we will hopefully see why when we read later patches.
Quoted 9 lines
object-file.c +0
-1@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,* give more context. */ static int hash_format_check_report(struct fsck_options *opts UNUSED, - const struct object_id *oid UNUSED,
object-file.c int repo_has_object_file(struct repository *r, · line 2478
Quoted 10 lines
object-file.c +6
-4- enum object_type object_type UNUSED, - enum fsck_msg_type msg_type UNUSED, - enum fsck_msg_id msg_id UNUSED, - const char *message) + const struct object_id *oid UNUSED, + enum object_type object_type UNUSED, + const char *ref_checked_name UNUSED, + enum fsck_msg_type msg_type UNUSED, + enum fsck_msg_id msg_id UNUSED, + const char *message)
That is somewhat annoying reindentation. What happened here?
Re: [GSoC][PATCH v10 03/10] fsck: add a unified interface for reporting fsck messages
Quoted 10 lines · Junio C Hamano
Quoted 7 lines · shejialuo
The static function "report" provided by "
fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "vfsck_report" by adding "checked_ref_name" parameter following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.
Quoted 9 lines
Like
strbuf_vinsertf(), it is a good idea to have "v" in the name of a function that takesva_list, butfsck_vreport()would probably be a better name here. Arguably, the originalreport()is misnamed (as a printf-like function that takes format string, it probably would have wanted to bereportf()instead), but unless we are fixing that at the same time, calling thisfsck_vreportf()would probably be too much. Consistently misnaming it by omitting the final "f" would be fine.
Yes,I will rename it to "fsck_vreport".
Quoted 4 lines
At this step it is still not clear if the previous step was really needed; you have this "v" thing that is designed to be usable by both reporting issues around objects and issues around refs, but we will hopefully see why when we read later patches.
From my perspective, I think we should put the previous commit after this commit. I agree with you that if we put it later, it will be much clearer and eaiser to understand.
Quoted 13 lines
Quoted 13 lines
object-file.c +0
-5@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,* give more context. */ static int hash_format_check_report(struct fsck_options *opts UNUSED, - const struct object_id *oid UNUSED, - enum object_type object_type UNUSED, - enum fsck_msg_type msg_type UNUSED, - enum fsck_msg_id msg_id UNUSED, -const char *message)
Quoted 8 lines
object-file.c int repo_has_object_file(struct repository *r, · line 2478
Quoted 6 lines
object-file.c +6
-0+ const struct object_id *oid UNUSED, + enum object_type object_type UNUSED, + const char *ref_checked_name UNUSED, + enum fsck_msg_type msg_type UNUSED, + enum fsck_msg_id msg_id UNUSED, + const char *message)That is somewhat annoying reindentation. What happened here?
The original code's indentation breaks. There are one more space in each line like the following:
static int hash_format_check_report(struct fsck_options *opts UNUSED,
const struct object_id *oid UNUSED)
...
I think I could fix this by the way.
PATCH v10 4/10 fsck: add refs-related error report function
In reply to shejialuoAdd refs-related options to the "fsck_options", create refs-specific "error_func" callback "fsck_refs_error_function".
"fsck_refs_error_function" will use the "oid" parameter. When the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.
Last, add "FSCK_REFS_OPTIONS_DEFAULT" and "FSCK_REFS_OPTIONS_STRICT" macros to create refs options easily.
--- fsck.c | 24 ++++++++++++++++++++++++ fsck.h | 14 ++++++++++++++ 2 files changed, 38 insertions(+)
fsck.c +24 -0
@@ -1252,6 +1252,30 @@ int fsck_objects_error_function(struct fsck_options *o,return 1 ; } +int fsck_refs_error_function(struct fsck_options *options UNUSED, +const struct object_id *oid, + enum object_type object_type UNUSED, +const char *checked_ref_name, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) +{ +struct strbuf sb =STRBUF_INIT ; +int ret =0 ; + +strbuf_addstr (&sb, checked_ref_name); +if (oid) +strbuf_addf (&sb," -> (%s)" ,oid_to_hex (oid)); + +if (msg_type ==FSCK_WARN ) +warning ("%s: %s" , sb.buf, message); +else + ret =error ("%s: %s" , sb.buf, message); + +strbuf_release (&sb); +return ret; +} +static int fsck_blobs (struct oidset *blobs_found,struct oidset *blobs_done,enum fsck_msg_id msg_missing,enum fsck_msg_id msg_type,struct fsck_options *options,const char *blob_type)
fsck.h +14 -0
@@ -135,6 +135,13 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); +int fsck_refs_error_function (struct fsck_options *options, +const struct object_id *oid, +enum object_type object_type, +const char *checked_ref_name, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);struct fsck_options {fsck_walk_func walk;
@@ -173,6 +180,13 @@ struct fsck_options { .gitattributes_done = OIDSET_INIT, \ .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ } +#define FSCK_REFS_OPTIONS_DEFAULT { \ + .error_func = fsck_refs_error_function, \ +} +#define FSCK_REFS_OPTIONS_STRICT { \ + .strict = 1, \ + .error_func = fsck_refs_error_function, \ +}/* descend in all linked child objects * the return value is:
Signature · 1 line
2.45.2
PATCH v10 5/10 refs: set up ref consistency check infrastructure
In reply to shejialuoThe interfaces defined in the ref_storage_be are carefully structured in semantic. It's organized as the five parts:
1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.
To keep consistent with the git-fsck(1), add a new interface named "fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.
Last, implement placeholder functions for each ref backends.
--- refs.c | 5 +++++ refs.h | 8 ++++++++ refs/debug.c | 11 +++++++++++ refs/files-backend.c | 15 ++++++++++++++- refs/packed-backend.c | 8 ++++++++ refs/refs-internal.h | 6 ++++++ refs/reftable-backend.c | 8 ++++++++ 7 files changed, 60 insertions(+), 1 deletion(-)
refs.c +5 -0
@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)return check_or_sanitize_refname (refname, flags,NULL ); } +int refs_fsck (struct ref_store *refs,struct fsck_options *o) +{ +return refs->be->fsck (refs, o); +} +void sanitize_refname_component (const char *refname,struct strbuf *out) {if (check_or_sanitize_refname (refname,REFNAME_ALLOW_ONELEVEL , out))
refs.h +8 -0
@@ -4,6 +4,7 @@#include "commit.h" #include "repository.h" +struct fsck_options ;struct object_id ;struct ref_store ;struct strbuf ;
@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat*/ int check_refname_format (const char *refname,int flags); +/* +* Check the reference database for consistency. Return 0 if refs and +* reflogs are consistent, and non-zero otherwise. The errors will be +* written to stderr. +*/ +int refs_fsck (struct ref_store *refs,struct fsck_options *o); +/* * Apply the rules from check_refname_format, but mutate the result until it * is acceptable, and place the result in "out".
refs/debug.c +11 -0
@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,return res; } +static int debug_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store; +int res = drefs->refs->be->fsck (drefs->refs, o); +trace_printf_key (&trace_refs,"fsck: %d\n" , res); +return res; +} +struct ref_storage_be refs_be_debug = { .name ="debug" , .init =NULL ,
@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = { .create_reflog = debug_create_reflog, .delete_reflog = debug_delete_reflog, .reflog_expire = debug_reflog_expire, + + .fsck = debug_fsck, };
refs/files-backend.c +14 -1
@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +static int files_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +int ret; +struct files_ref_store *refs = +files_downcast (ref_store,REF_STORE_READ ,"fsck" ); + + ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); +return ret; +} +struct ref_storage_be refs_be_files = { .name ="files" , .init = files_ref_store_init,
@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = { .reflog_exists = files_reflog_exists, .create_reflog = files_create_reflog, .delete_reflog = files_delete_reflog, - .reflog_expire = files_reflog_expire + .reflog_expire = files_reflog_expire, + + .fsck = files_fsck, };
refs/packed-backend.c +8 -0
@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_sreturn empty_ref_iterator_begin (); } +static int packed_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_packed = { .name ="packed" , .init = packed_ref_store_init,
@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = { .create_reflog = NULL, .delete_reflog = NULL, .reflog_expire = NULL, + + .fsck = packed_fsck, };
refs/refs-internal.h +6 -0
@@ -4,6 +4,7 @@#include "refs.h" #include "iterator.h" +struct fsck_options ;struct ref_transaction ;/*
@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,typedef int read_symbolic_ref_fn (struct ref_store *ref_store,const char *refname,struct strbuf *referent); +typedef int fsck_fn (struct ref_store *ref_store, +struct fsck_options *o); +struct ref_storage_be {const char *name;ref_store_init_fn *init;
@@ -677,6 +681,8 @@ struct ref_storage_be {create_reflog_fn *create_reflog;delete_reflog_fn *delete_reflog;reflog_expire_fn *reflog_expire; + +fsck_fn *fsck; };extern struct ref_storage_be refs_be_files;
refs/reftable-backend.c +8 -0
@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,return ret; } +static int reftable_be_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_reftable = { .name ="reftable" , .init = reftable_be_init,
@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = { .create_reflog = reftable_be_create_reflog, .delete_reflog = reftable_be_delete_reflog, .reflog_expire = reftable_be_reflog_expire, + + .fsck = reftable_be_fsck, };
Signature · 1 line
2.45.2
PATCH v10 6/10 builtin/refs: add verify subcommand and verbose_refs for "fsck_options"
In reply to shejialuoIntroduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency and also this subcommand will be used as the entry point of checking refs for "git-fsck(1)". Last, add "verbose_refs" field into "fsck_options" to indicate whether we should print verbose messages when checking refs consistency.
--- Documentation/git-refs.txt | 13 +++++++++++ builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++ fsck.h | 1 + 3 files changed, 58 insertions(+)
Documentation/git-refs.txt +13 -0
@@ -10,6 +10,7 @@ SYNOPSIS -------- [verse] 'git refs migrate' --ref-format=<format> [--dry-run] +'git refs verify' [--strict] [--verbose] DESCRIPTION -----------
@@ -22,6 +23,9 @@ COMMANDS migrate:: Migrate ref store between different formats. +verify:: + Verify reference database consistency. + OPTIONS -------
@@ -39,6 +43,15 @@ include::ref-storage-format.txt[] can be used to double check that the migration works as expected before performing the actual migration. +The following options are specific to 'git refs verify': + +--strict:: + Enable more strict checking, every WARN severity for the `Fsck Messages` + be seen as ERROR. See linkgit:git-fsck[1]. + +--verbose:: + When verifying the reference database consistency, be chatty. + KNOWN LIMITATIONS -----------------
builtin/refs.c +44 -0
69 lines not on this page · raw message
fsck.h +1 -0
@@ -147,6 +147,7 @@ struct fsck_options {fsck_walk_func walk;fsck_error error_func; unsigned strict:1; + unsigned verbose_refs:1;enum fsck_msg_type *msg_type;struct oidset skip_oids;struct oidset gitmodules_found;
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v10 06/10] builtin/refs: add verify subcommand and verbose_refs for "fsck_options"
Quoted 1 line · shejialuo
Subject: Re: [GSoC][PATCH v10 06/10] builtin/refs: add verify subcommand and
verbose_refsfor "fsck_options"
Just saying
git refs: add verify subcommand
would be clearer. If you really want to talk about two modes, you could say
git refs: add "verify [--strict|--verbose]" subcommand
but that may be too much.
Quoted 5 lines
Introduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency and also this subcommand will be used as the entry point of checking refs for "git-fsck(1)". Last, add "
verbose_refs" field into "fsck_options" to indicate whether we should print verbose messages when checking refs consistency.
Is there a reason why this has to be verbose_refs and not a simple verbose bit? When people see how it is useful to ask for the verbose output while checking refs, wouldn't people wish to add the same "--verbose" support while checking objects, and at that point, wouldn't it be awkward to add verbose_objs member to the struct and having to flip both bits on?
Quoted 22 lines
Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- Documentation/git-refs.txt | 13 +++++++++++ builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++ fsck.h | 1 + 3 files changed, 58 insertions(+)Documentation/git-refs.txt +1
-0@@ -10,6 +10,7 @@ SYNOPSIS -------- [verse] 'git refs migrate' --ref-format=<format> [--dry-run] +'git refs verify' [--strict] [--verbose] DESCRIPTION -----------
Documentation/git-refs.txt COMMANDS · line 28
Quoted 6 lines
Documentation/git-refs.txt +3
-0@@ -22,6 +23,9 @@ COMMANDS migrate:: Migrate ref store between different formats. +verify:: + Verify reference database consistency. +
The error reporting function for refs consistency check was still about reporting a problem for a single ref. I am wondering how consistency violations that are not about a single ref should be handled. For example, if refs/packed-backend.c:packed_fsck() finds that the file is not sorted properly or has some unparseable garbage in it, it is not something you can report as "refs/heads/main is broken", but those who are interested in seeing the "reference database consistency" verified, it is very much what they want the tool to notice. How would detection of such a breakage that is not attributed to a single ref fit in this "ref consistency check infrastructure" that was introduced by [05/10]?
Documentation/git-refs.txt COMMANDS · line 31
Quoted 3 lines
Documentation/git-refs.txt +3
-0+ argc = parse_options(argc, argv, prefix, options, verify_usage, 0); + if (argc) + usage(_("too many arguments"));
I do not think we want to change this line in this topic, but because I noticed that the issue is widespread, let me make a note here that we may want to clean up all the commands that give this message as a #leftoverbit item:
$ git cmd foo baz
usage: too many argumentsis very unfriendly in that it is not immediately obvious to users which arguments are excess. Should they have given "git cmd<RET>"? Is it "git cmd foo" that does not take any argument?
If you said something like
$ git refs verify baz
error: 'git refs verify' takes no argumentsor even
$ git refs verify baz
error: unknown argument 'baz' given to 'git refs verify'it would be much more helpful.
Thanks.
Re: [GSoC][PATCH v10 06/10] builtin/refs: add verify subcommand and verbose_refs for "fsck_options"
Quoted 6 lines · Junio C Hamano
Quoted 1 line · shejialuo
Subject: Re: [GSoC][PATCH v10 06/10] builtin/refs: add verify subcommand and
verbose_refsfor "fsck_options"Just saying
Quoted 9 lines
git refs: add verify subcommand
would be clearer. If you really want to talk about two modes, you could say
git refs: add "verify [--strict|--verbose]" subcommand
but that may be too much.
Thanks, I will change in the next version.
Quoted 6 lines
Quoted 5 lines
Introduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency and also this subcommand will be used as the entry point of checking refs for "git-fsck(1)". Last, add "
verbose_refs" field into "fsck_options" to indicate whether we should print verbose messages when checking refs consistency.
Quoted 7 lines
Is there a reason why this has to be
verbose_refsand not a simple verbose bit? When people see how it is useful to ask for the verbose output while checking refs, wouldn't people wish to add the same "--verbose" support while checking objects, and at that point, wouldn't it be awkward to addverbose_objsmember to the struct and having to flip both bits on?
Actually, this is really what I thought. I just want to provide more find-grained control here. However, when I implemented the code, I also felt awkward about this. I can't find the balance here.
I will improve this in the next version.
Quoted 25 lines
Quoted 25 lines
Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- Documentation/git-refs.txt | 13 +++++++++++ builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++ fsck.h | 1 + 3 files changed, 58 insertions(+)Documentation/git-refs.txt +1
-0@@ -10,6 +10,7 @@ SYNOPSIS -------- [verse] 'git refs migrate' --ref-format=<format> [--dry-run] +'git refs verify' [--strict] [--verbose] DESCRIPTION -----------@@ -22,6 +23,9 @@ COMMANDS migrate:: Migrate ref store between different formats.
Quoted 16 lines
Documentation/git-refs.txt COMMANDS · line 28
Quoted 3 lines
Documentation/git-refs.txt +3
-0+verify:: + Verify reference database consistency. +The error reporting function for refs consistency check was still about reporting a problem for a single ref. I am wondering how consistency violations that are not about a single ref should be handled. For example, if refs/packed-backend.c:packed_fsck() finds that the file is not sorted properly or has some unparseable garbage in it, it is not something you can report as "refs/heads/main is broken", but those who are interested in seeing the "reference database consistency" verified, it is very much what they want the tool to notice. How would detection of such a breakage that is not attributed to a single ref fit in this "ref consistency check infrastructure" that was introduced by [05/10]?
Yes, I didn't consider other cases. Although I have said in the subject that this series is to set up the infrastructure of fscking refs. It's a little hard for me to set up a perfect "fsck_refs_report" at the moment.
As you said, I think currently I should consider about the packed-refs in this series. I will find a way to achieve this in the next version. Well, I could say I intentionally ignored this problem. But we should face the problem directly.
Really thanks.
Quoted 21 lines
Documentation/git-refs.txt COMMANDS · line 31
Quoted 3 lines
Documentation/git-refs.txt +3
-0+ argc = parse_options(argc, argv, prefix, options, verify_usage, 0); + if (argc) + usage(_("too many arguments"));I do not think we want to change this line in this topic, but because I noticed that the issue is widespread, let me make a note here that we may want to clean up all the commands that give this message as a #leftoverbit item:
$ git cmd foo baz usage: too many argumentsis very unfriendly in that it is not immediately obvious to users which arguments are excess. Should they have given "git cmd<RET>"? Is it "git cmd foo" that does not take any argument?
If you said something like
$ git refs verify baz error: 'git refs verify' takes no arguments
Quoted 7 lines
or even
$ git refs verify baz error: unknown argument 'baz' given to 'git refs verify'it would be much more helpful.
I will improve this in the next version.
Quoted 2 lines
Thanks.
PATCH v10 7/10 builtin/fsck: add `git-refs verify` child process
In reply to shejialuoIntroduce a new function "fsck_refs" that initializes and runs a child process to execute the "git-refs verify" command.
--- builtin/fsck.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
builtin/fsck.c +17 -0
@@ -897,6 +897,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)return res; } +static void fsck_refs (void ) +{ +struct child_process refs_verify =CHILD_PROCESS_INIT ; +child_process_init (&refs_verify); + refs_verify.git_cmd =1 ; +strvec_pushl (&refs_verify.args,"refs" ,"verify" ,NULL ); +if (verbose) +strvec_push (&refs_verify.args,"--verbose" ); +if (check_strict) +strvec_push (&refs_verify.args,"--strict" ); + +if (run_command (&refs_verify)) + errors_found |=ERROR_REFS ; +} +static char const *const fsck_usage[] = { N_("git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\n" " [--[no-]full] [--strict] [--verbose] [--lost-found]\n"
@@ -1066,6 +1081,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)check_connectivity (); +fsck_refs (); +if (the_repository->settings.core_commit_graph) {struct child_process commit_graph_verify =CHILD_PROCESS_INIT ;
Signature · 1 line
2.45.2
PATCH v10 8/10 files-backend: add unified interface for refs scanning
In reply to shejialuoFor refs and reflogs, we need to scan its corresponding directories to check every regular file or symbolic link which shares the same pattern. Introduce a unified interface for scanning directories for files-backend.
--- refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 76 insertions(+), 1 deletion(-)
refs/files-backend.c +76 -1
@@ -6,6 +6,7 @@#include "../gettext.h" #include "../hash.h" #include "../hex.h" +#include "../fsck.h" #include "../refs.h" #include "refs-internal.h" #include "ref-cache.h"
@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +/* +* For refs and reflogs, they share a unified interface when scanning +* the whole directory. This function is used as the callback for each +* regular file or symlink in the directory. +*/ +typedef int (*files_fsck_refs_fn )(struct fsck_options *o, +const char *gitdir, +const char *refs_check_dir, +struct dir_iterator *iter); + +static int files_fsck_refs_dir (struct ref_store *ref_store, +struct fsck_options *o, +const char *refs_check_dir, +files_fsck_refs_fn *fsck_refs_fns) +{ +const char *gitdir = ref_store->gitdir; +struct strbuf sb =STRBUF_INIT ; +struct dir_iterator *iter; +int iter_status; +int ret =0 ; + +strbuf_addf (&sb,"%s/%s" , gitdir, refs_check_dir); + + iter =dir_iterator_begin (sb.buf,0 ); + +if (!iter) { + ret =error_errno ("cannot open directory %s" , sb.buf); + goto out; + } + +while ((iter_status =dir_iterator_advance (iter)) ==ITER_OK ) { +if (S_ISDIR (iter->st.st_mode)) { +continue ; + }else if (S_ISREG (iter->st.st_mode) || +S_ISLNK (iter->st.st_mode)) { +if (o->verbose_refs) +fprintf_ln (stderr,"Checking %s/%s" , + refs_check_dir, iter->relative_path); +for (size_t i =0 ; fsck_refs_fns[i]; i++) { +if (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter)) + ret =-1 ; + } + }else { + ret =error (_ ("unexpected file type for '%s'" ), + iter->basename); + } + } + +if (iter_status !=ITER_DONE ) + ret =error (_ ("failed to iterate over '%s'" ), sb.buf); + +out: +strbuf_release (&sb); +return ret; +} + +static int files_fsck_refs (struct ref_store *ref_store, +struct fsck_options *o) +{ +int ret; +files_fsck_refs_fn fsck_refs_fns[]= { +NULL + }; + +if (o->verbose_refs) +fprintf_ln (stderr,"Checking references consistency" ); + + ret =files_fsck_refs_dir (ref_store, o,"refs" , fsck_refs_fns); + +return ret; +} +static int files_fsck (struct ref_store *ref_store,struct fsck_options *o) {
@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,struct files_ref_store *refs =files_downcast (ref_store,REF_STORE_READ ,"fsck" ); - ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); + ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o) + |files_fsck_refs (ref_store, o); +return ret; }
Signature · 1 line
2.45.2
PATCH v10 9/10 fsck: add ref name check for files backend
In reply to shejialuoThe git-fsck(1) only implicitly checks the reference, it does not fully check refs with bad format name such as standalone "@" and name ending with ".lock".
In order to provide such checks, add a new fsck message id "badRefName" with default ERROR type. Use existing "check_refname_format" to explicit check the ref name. And add a new unit test to verify the functionality.
--- Documentation/fsck-msgids.txt | 3 + fsck.h | 1 + refs/files-backend.c | 20 +++++++ t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++ 4 files changed, 125 insertions(+) create mode 100755 t/t0602-reffiles-fsck.sh
Documentation/fsck-msgids.txt +3 -0
@@ -19,6 +19,9 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefName`:: + (ERROR) A ref has a bad name. + `badTagName`:: (INFO) A tag has an invalid format.
fsck.h +1 -0
@@ -31,6 +31,7 @@ enum fsck_msg_type { FUNC(BAD_NAME, ERROR) \ FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ + FUNC(BAD_REF_NAME, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
refs/files-backend.c +20 -0
@@ -3419,6 +3419,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,const char *refs_check_dir, struct dir_iterator *iter); +static int files_fsck_refs_name (struct fsck_options *o, + const char *gitdir UNUSED, +const char *refs_check_dir, +struct dir_iterator *iter) +{ +struct strbuf sb =STRBUF_INIT ; +int ret =0 ; + +if (check_refname_format (iter->basename,REFNAME_ALLOW_ONELEVEL )) { +strbuf_addf (&sb,"%s/%s" , refs_check_dir, iter->relative_path); + ret =fsck_refs_report (o,NULL , sb.buf, +FSCK_MSG_BAD_REF_NAME , +"invalid refname format" ); + } + +strbuf_release (&sb); +return ret; +} + static int files_fsck_refs_dir(struct ref_store *ref_store, struct fsck_options *o, const char *refs_check_dir,
@@ -3470,6 +3489,7 @@ static int files_fsck_refs(struct ref_store *ref_store, {int ret;files_fsck_refs_fn fsck_refs_fns[]= { + files_fsck_refs_name,NULL };
t/t0602-reffiles-fsck.sh new file +101 -0
101 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v10 10/10 fsck: add ref content check for files backend
In reply to shejialuoEnhance the git-fsck(1) command by adding a check for reference content in the files backend. The new functionality ensures that symrefs, real symbolic link and regular refs are validated correctly.
In order to check the trailing content of the regular refs, add a new parameter trailing to parse_loose_ref_contents.
For symrefs, parse_loose_ref_contents will set the "referent". However, symbolic link could be either absolute or relative. Use "strbuf_add_real_path" to read the symbolic link and convert the relative path to absolute path. Then use "skip_prefix" to make it align with symref "referent".
Thus, the symrefs and symbolic links could share the same interface. Add a new function "files_fsck_symref_target" which aims at checking the following things:
1. whether the pointee is under the refs/ directory.
2. whether the pointee name is correct.
3. whether the pointee path is a wrong type in filesystem.
Last, add the following FSCK MESSAGEs:
1. "badRefContent(ERROR)": A ref has a bad content
2. "badSymrefPointee(ERROR)": The pointee of a symref is bad.
3. "trailingRefContent(WARN)": A ref content has trailing contents.
--- Documentation/fsck-msgids.txt | 9 +++ fsck.h | 3 + refs.c | 2 +- refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++- refs/refs-internal.h | 5 +- t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++ 6 files changed, 269 insertions(+), 5 deletions(-)
Documentation/fsck-msgids.txt +9 -0
@@ -19,9 +19,15 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefContent`:: + (ERROR) A ref has a bad content. + `badRefName`:: (ERROR) A ref has a bad name. +`badSymrefPointee`:: + (ERROR) The pointee of a symref is bad. + `badTagName`:: (INFO) A tag has an invalid format.
@@ -167,6 +173,9 @@ `nullSha1`:: (WARN) Tree contains entries pointing to a null sha1. +`trailingRefContent`:: + (WARN) A ref content has trailing contents. + `treeNotSorted`:: (ERROR) A tree is not properly sorted.
fsck.h +3 -0
@@ -32,6 +32,8 @@ enum fsck_msg_type { FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ FUNC(BAD_REF_NAME, ERROR) \ + FUNC(BAD_REF_CONTENT, ERROR) \ + FUNC(BAD_SYMREF_POINTEE, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
@@ -72,6 +74,7 @@ enum fsck_msg_type { FUNC(HAS_DOTDOT, WARN) \ FUNC(HAS_DOTGIT, WARN) \ FUNC(NULL_SHA1, WARN) \ + FUNC(TRAILING_REF_CONTENT, WARN) \ FUNC(ZERO_PADDED_FILEMODE, WARN) \ FUNC(NUL_IN_COMMIT, WARN) \FUNC (LARGE_PATHNAME ,WARN ) \
refs.c +1 -1
@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store, } result =parse_loose_ref_contents (content.buf, oid, referent, type, - failure_errno); + failure_errno,NULL ); done:strbuf_release (&full_path);
refs/files-backend.c +143 -2
181 lines not on this page · raw message
refs/refs-internal.h +3 -2
@@ -709,11 +709,12 @@ struct ref_store {/* * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for -* invalid contents. +* invalid contents. Also *trailing is set to the first character after the +* refname or NULL if the referent is not empty. */ int parse_loose_ref_contents (const char *buf,struct object_id *oid,struct strbuf *referent,unsigned int *type, -int *failure_errno); +int *failure_errno,const char **trailing);/* * Fill in the generic part of refs and add it to our collection of
t/t0602-reffiles-fsck.sh +110 -0
114 lines not on this page · raw message
Signature · 1 line
2.45.2
v11 cover ref consistency check infra setup
In reply to shejialuoHi All:
This version handles the following problems:
1. Reorder the commit message to let "fsck: rename objects-related fsck error functions" behind the "fsck: add a unified interface for reporting fsck messages".
2. Rename "vfsck_report" to "fsck_vreport" to be align with the codebase.
3. Make the "git-refs verify" commit message clean, and add user-friendly error message here.
4. Rename "verbose_refs" to "verbose" field in "fsck_options".
And the most important change in this version is to enhance the "fsck_refs_report" function. After some investigations, there are the following situations where we should handle when checking ref consistency.
1. When checking loose refs and reflofs, we only need the checkee information, because they are standalone files.
2. When checking packed-refs, we should check the packed-refs itself, for example whether it is sorted or there are some garbage trailing contents. However, we should also check each ref (sub_checkee) in the file.
3. When checking reftable refs, we need to check the binary file, however, I does not truly understand the principle of reftable refs. But we could still use the idea like 2 case.
By the above statements, I change the "fsck_refs_report" shown as below:
int fsck_refs_error_function(struct fsck_options *options UNUSED,
const struct object_id *oid, enum object_type object_type UNUSED, const char *ref_checkee, const char *sub_ref_checkee, enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED, const char *message)
{
struct strbuf sb = STRBUF_INIT;
int ret = 0;
if (sub_ref_checkee)
strbuf_addf(&sb, "%s.%s", ref_checkee, sub_ref_checkee);
else
strbuf_addstr(&sb, ref_checkee);
if (oid)
strbuf_addf(&sb, " -> (%s)", oid_to_hex(oid));
if (msg_type == FSCK_WARN)
warning("%s: %s", sb.buf, message);
else
ret = error("%s: %s", sb.buf, message);
strbuf_release(&sb);
return ret;
}It could provide the following report messages:
1. "ref_checkee": "fsck error name": "user message".
2. "ref_checkee.sub_ref_checkee": "fsck error name": "user message".
3. "ref_checkee -> (oid hex)": "fsck error name": "user message".
4. "ref_checkee.sub_ref_checkee -> (oid hex)": "fsck error name": "user message".
shejialuo (10):
fsck: rename "skiplist" to "skip_oids"
fsck: add a unified interface for reporting fsck messages
fsck: rename objects-related fsck error functions
fsck: add refs-related error report function
refs: set up ref consistency check infrastructure git refs: add verify subcommand
builtin/fsck: add git-refs verify child process
files-backend: add unified interface for refs scanning
fsck: add ref name check for files backend
fsck: add ref content check for files backend
Documentation/fsck-msgids.txt | 12 ++ Documentation/git-refs.txt | 13 ++ builtin/fsck.c | 33 ++++- builtin/mktag.c | 2 + builtin/refs.c | 44 ++++++ fsck.c | 118 +++++++++++++--- fsck.h | 69 ++++++--- object-file.c | 12 +- refs.c | 7 +- refs.h | 8 ++ refs/debug.c | 11 ++ refs/files-backend.c | 255 +++++++++++++++++++++++++++++++++- refs/packed-backend.c | 8 ++ refs/refs-internal.h | 11 +- refs/reftable-backend.c | 8 ++ t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++ 16 files changed, 766 insertions(+), 56 deletions(-) create mode 100755 t/t0602-reffiles-fsck.sh
Range-diff against v10:
1: e044f933de = 1: a69705b777 fsck: rename "skiplist" to "skip_oids" 3: df4837e960 ! 2: 1ef1036348 fsck: add a unified interface for reporting fsck messages
@@ Commit message
error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which
- can report either objects or refs, create a new function "vfsck_report"
- by adding "checked_ref_name" parameter following the "report" prototype.
- Instead of using "...", provide "va_list" to allow more flexibility.
+ can report either objects or refs, create a new function "fsck_vreport"
+ following the "report" prototype. Instead of using "...", provide
+ "va_list" to allow more flexibility.
- Like "report", the "vfsck_report" function will use "error_func"
+ When checking loose refs and reflogs, we only need to pass the checked
+ name to the fsck error report function. However, for packed-refs and
+ reftable refs, we need to check both the consistency of the file itself
+ and the refs or reflogs contained in the file. In order to provide above
+ checks, add two parameters "ref_checkee" and "sub_ref_checkee" in
+ "fsck_vreport" function.
+
+ Like "report", the "fsck_vreport" function will use "error_func"
registered in "fsck_options" to report customized messages. Change
- "error_func" prototype to align with the new "vfsck_report".
+ "error_func" prototype to align with the new "fsck_vreport".
- Then, change "report" function to use "vfsck_report" to report objects
+ Then, change "report" function to use "fsck_vreport" to report objects
related messages. Add a new function called "fsck_refs_report" to use
- "vfsck_report" to report refs related messages.
+ "fsck_vreport" to report refs related messages.
@@ Commit message
## builtin/fsck.c ##
@@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)
- static int fsck_objects_error_func(struct fsck_options *o UNUSED,
- const struct object_id *oid,
- enum object_type object_type,
-+ const char *checked_ref_name UNUSED,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+ static int fsck_error_func(struct fsck_options *o UNUSED,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const char *ref_checkee UNUSED,
++ const char *sub_ref_checkee UNUSED,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
## builtin/mktag.c ##
@@ builtin/mktag.c: static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT;
static int mktag_fsck_error_func(struct fsck_options *o UNUSED,
const struct object_id *oid UNUSED,
enum object_type object_type UNUSED,
-+ const char *checked_ref_name UNUSED,
++ const char *ref_checkee UNUSED,
++ const char *sub_ref_checkee UNUSED,
enum fsck_msg_type msg_type,
enum fsck_msg_id msg_id UNUSED,
const char *message)
@@ fsck.c: static int object_on_skiplist(struct fsck_options *opts,
+ * It will get the current msg error type and call the error_func callback
+ * which is registered in the "fsck_options" struct.
+ */
-+static int vfsck_report(struct fsck_options *options,
++static int fsck_vreport(struct fsck_options *options,
+ const struct object_id *oid,
+ enum object_type object_type,
-+ const char *checked_ref_name,
++ const char *ref_checkee,
++ const char *sub_ref_checkee,
+ enum fsck_msg_id msg_id, const char *fmt, va_list ap)
{
- va_list ap;
@@ fsck.c: static int report(struct fsck_options *options,
- va_start(ap, fmt);
- strbuf_vaddf(&sb, fmt, ap);
-- result = options->error_func(options, oid, object_type,
+ va_copy(ap_copy, ap);
+ strbuf_vaddf(&sb, fmt, ap_copy);
-+ result = options->error_func(options, oid, object_type, checked_ref_name,
+ result = options->error_func(options, oid, object_type,
++ ref_checkee, sub_ref_checkee,
msg_type, msg_id, sb.buf);
strbuf_release(&sb);
va_end(ap);
@@ fsck.c: static int report(struct fsck_options *options,
+{
+ va_list ap;
+ int result;
++
+ va_start(ap, fmt);
-+ result = vfsck_report(options, oid, object_type, NULL,
++ result = fsck_vreport(options, oid, object_type, NULL, NULL,
+ msg_id, fmt, ap);
+ va_end(ap);
++
+ return result;
+}
+
-+
-+
+int fsck_refs_report(struct fsck_options *options,
+ const struct object_id *oid,
-+ const char *checked_ref_name,
++ const char *ref_checkee,
++ const char *sub_ref_checkee,
+ enum fsck_msg_id msg_id, const char *fmt, ...)
+{
+ va_list ap;
+ int result;
+ va_start(ap, fmt);
-+ result = vfsck_report(options, oid, OBJ_NONE,
-+ checked_ref_name, msg_id, fmt, ap);
++ result = fsck_vreport(options, oid, OBJ_NONE, ref_checkee, sub_ref_checkee,
++ msg_id, fmt, ap);
+ va_end(ap);
+ return result;
+}
@@ fsck.c: static int report(struct fsck_options *options,
{
if (!options->object_names)
@@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,
- int fsck_objects_error_function(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type UNUSED,
-+ const char *checked_ref_name UNUSED,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+ int fsck_error_function(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type UNUSED,
++ const char *ref_checkee UNUSED,
++ const char *sub_ref_checkee UNUSED,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
@@ fsck.c: int git_fsck_config(const char *var, const char *value,
- int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type,
-+ const char *checked_ref_name,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message)
-@@ fsck.c: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const char *ref_checkee,
++ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message)
+@@ fsck.c: int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
puts(oid_to_hex(oid));
return 0;
}
-- return fsck_objects_error_function(o, oid, object_type,
-+ return fsck_objects_error_function(o, oid, object_type, checked_ref_name,
- msg_type, msg_id, message);
+- return fsck_error_function(o, oid, object_type, msg_type, msg_id, message);
++ return fsck_error_function(o, oid, object_type, ref_checkee,
++ sub_ref_checkee, msg_type, msg_id, message);
}
## fsck.h ##
@@ fsck.h: int is_valid_msg_type(const char *msg_id, const char *msg_type);
+ */
typedef int (*fsck_error)(struct fsck_options *o,
const struct object_id *oid, enum object_type object_type,
-+ const char *checked_ref_name,
++ const char *ref_checkee, const char *sub_ref_checkee,
enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message);
- int fsck_objects_error_function(struct fsck_options *o,
- const struct object_id *oid, enum object_type object_type,
-+ const char *checked_ref_name,
- enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
- const char *message);
- int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type,
-+ const char *checked_ref_name,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message);
+ int fsck_error_function(struct fsck_options *o,
+ const struct object_id *oid, enum object_type object_type,
++ const char *ref_checkee, const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
+ const char *message);
+ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const char *ref_checkee,
++ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message);
@@ fsck.h: int fsck_tag_standalone(const struct object_id *oid, const char *buffer,
*/
int fsck_finish(struct fsck_options *options);
@@ fsck.h: int fsck_tag_standalone(const struct object_id *oid, const char *buffer,
+/*
+ * Report an error or warning for refs.
+ */
-+__attribute__((format (printf, 5, 6)))
++__attribute__((format (printf, 6, 7)))
+int fsck_refs_report(struct fsck_options *options,
+ const struct object_id *oid,
-+ const char *checked_ref_name,
++ const char *ref_checkee,
++ const char *sub_ref_checkee,
+ enum fsck_msg_id msg_id,
+ const char *fmt, ...);
+
@@ object-file.c: int repo_has_object_file(struct repository *r,
- const char *message)
+ const struct object_id *oid UNUSED,
+ enum object_type object_type UNUSED,
-+ const char *ref_checked_name UNUSED,
++ const char *ref_checkee UNUSED,
++ const char *sub_ref_checkee UNUSED,
+ enum fsck_msg_type msg_type UNUSED,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
2: 73a7c53a23 ! 3: d17cf6166e fsck: rename objects-related fsck error functions
@@ Commit message
fsck: rename objects-related fsck error functions
The names of objects-related fsck error functions are general. It's OK
- when there is only object database check. However, we are going to
- introduce refs database check. In order to avoid ambiguity, rename
- objects-related fsck error functions to explicitly indicate these
+ when there is only object database check. However, we have introduced
+ refs database check report function. To avoid ambiguity, rename
+ object-related fsck error functions to explicitly indicate these
functions are used to report objects-related messages.
@@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)
-static int fsck_error_func(struct fsck_options *o UNUSED,
- const struct object_id *oid,
- enum object_type object_type,
+- const char *ref_checkee UNUSED,
+- const char *sub_ref_checkee UNUSED,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+static int fsck_objects_error_func(struct fsck_options *o UNUSED,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const char *ref_checkee UNUSED,
++ const char *sub_ref_checkee UNUSED,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
@@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,
-int fsck_error_function(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type UNUSED,
+- const char *ref_checkee UNUSED,
+- const char *sub_ref_checkee UNUSED,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+int fsck_objects_error_function(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type UNUSED,
++ const char *ref_checkee UNUSED,
++ const char *sub_ref_checkee UNUSED,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
@@ fsck.c: int git_fsck_config(const char *var, const char *value,
-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type,
+- const char *ref_checkee,
+- const char *sub_ref_checkee,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message)
+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const char *ref_checkee,
++ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message)
@@ fsck.c: int git_fsck_config(const char *var, const char *value,
puts(oid_to_hex(oid));
return 0;
}
-- return fsck_error_function(o, oid, object_type, msg_type, msg_id, message);
-+ return fsck_objects_error_function(o, oid, object_type,
-+ msg_type, msg_id, message);
+- return fsck_error_function(o, oid, object_type, ref_checkee,
+- sub_ref_checkee, msg_type, msg_id, message);
++ return fsck_objects_error_function(o, oid, object_type, ref_checkee,
++ sub_ref_checkee, msg_type, msg_id,
++ message);
}
## fsck.h ##
@@ fsck.h: typedef int (*fsck_error)(struct fsck_options *o,
-int fsck_error_function(struct fsck_options *o,
- const struct object_id *oid, enum object_type object_type,
+- const char *ref_checkee, const char *sub_ref_checkee,
- enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
- const char *message);
-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type,
+- const char *ref_checkee,
+- const char *sub_ref_checkee,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message);
+int fsck_objects_error_function(struct fsck_options *o,
+ const struct object_id *oid, enum object_type object_type,
++ const char *ref_checkee, const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
+ const char *message);
+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const char *ref_checkee,
++ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message);
4: c8eb01c987 ! 4: ee17c0835b fsck: add refs-related error report function
@@ Metadata
## Commit message ##
fsck: add refs-related error report function
- Add refs-related options to the "fsck_options", create refs-specific
- "error_func" callback "fsck_refs_error_function".
+ Create refs-specific "error_func" callback "fsck_refs_error_function"
+ which could provide the following report messages.
- "fsck_refs_error_function" will use the "oid" parameter. When the caller
+ 1. "ref_checkee": "fsck error name": "user message".
+ 2. "ref_checkee.sub_ref_checkee": "fsck error name": "user message".
+ 3. "ref_checkee -> (oid hex)": "fsck error name": "user message".
+ 4. "ref_checkee.sub_ref_checkee -> (pid hex)": "fsck error name": "user
+ message".
+
+ "fsck_refs_error_function" uses the "ref_checkee" and "sub_ref_checkee"
+ parameters to indicate the information of the checked refs. For loose
+ ref and reflog, it only uses the "ref_checkee" parameter. For packed
+ refs and reftable refs, when checking the consistency of the file
+ itself, it still only uses "ref_checkee" parameter. However, when
+ checking the consistency of the ref or reflog contained in the file, it
+ will use "sub_ref_checkee" parameter to indicate that we are not
+ checking the file but the incorporated ref or reflog.
+
+ "fsck_refs_error_function" will use the "oid" parameter if the caller
passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.
@@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,
+int fsck_refs_error_function(struct fsck_options *options UNUSED,
+ const struct object_id *oid,
+ enum object_type object_type UNUSED,
-+ const char *checked_ref_name,
++ const char *ref_checkee,
++ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
@@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,
+ struct strbuf sb = STRBUF_INIT;
+ int ret = 0;
+
-+ strbuf_addstr(&sb, checked_ref_name);
++ if (sub_ref_checkee)
++ strbuf_addf(&sb, "%s.%s", ref_checkee, sub_ref_checkee);
++ else
++ strbuf_addstr(&sb, ref_checkee);
++
+ if (oid)
+ strbuf_addf(&sb, " -> (%s)", oid_to_hex(oid));
+
@@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *
+int fsck_refs_error_function(struct fsck_options *options,
+ const struct object_id *oid,
+ enum object_type object_type,
-+ const char *checked_ref_name,
++ const char *ref_checkee,
++ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message);
5: e4085df496 = 5: 4718ba7ddc refs: set up ref consistency check infrastructure
6: 497f224bed ! 6: 76163fb5d9 builtin/refs: add verify subcommand and verbose_refs for "fsck_options"
@@ Metadata
Author: shejialuo <[email protected]>
## Commit message ##
- builtin/refs: add verify subcommand and verbose_refs for "fsck_options"
+ git refs: add verify subcommand
Introduce a new subcommand "verify" in git-refs(1) to allow the user to
check the reference database consistency and also this subcommand will
be used as the entry point of checking refs for "git-fsck(1)". Last, add
- "verbose_refs" field into "fsck_options" to indicate whether we should
- print verbose messages when checking refs consistency.
+ "verbose" field into "fsck_options" to indicate whether we should print
+ verbose messages when checking refs and objects consistency.
Mentored-by: Patrick Steinhardt <[email protected]>
Mentored-by: Karthik Nayak <[email protected]>
@@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c
+
+ argc = parse_options(argc, argv, prefix, options, verify_usage, 0);
+ if (argc)
-+ usage(_("too many arguments"));
++ usage(_("'git refs verify' takes no arguments"));
+
+ if (verbose)
-+ fsck_refs_options.verbose_refs = 1;
++ fsck_refs_options.verbose = 1;
+ if (strict)
+ fsck_refs_options.strict = 1;
+
@@ fsck.h: struct fsck_options {
fsck_walk_func walk;
fsck_error error_func;
unsigned strict:1;
-+ unsigned verbose_refs:1;
++ unsigned verbose:1;
enum fsck_msg_type *msg_type;
struct oidset skip_oids;
struct oidset gitmodules_found;
7: 86a14c7b43 = 7: 27f766fb8e builtin/fsck: add `git-refs verify` child process
8: daedb80b47 ! 8: e2ab45ec9f files-backend: add unified interface for refs scanning
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
+ continue;
+ } else if (S_ISREG(iter->st.st_mode) ||
+ S_ISLNK(iter->st.st_mode)) {
-+ if (o->verbose_refs)
++ if (o->verbose)
+ fprintf_ln(stderr, "Checking %s/%s",
+ refs_check_dir, iter->relative_path);
+ for (size_t i = 0; fsck_refs_fns[i]; i++) {
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
+ NULL
+ };
+
-+ if (o->verbose_refs)
++ if (o->verbose)
+ fprintf_ln(stderr, "Checking references consistency");
+
+ ret = files_fsck_refs_dir(ref_store, o, "refs", fsck_refs_fns);
9: c36d588e4f ! 9: cd438fb56d fsck: add ref name check for files backend
@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,
+
+ if (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {
+ strbuf_addf(&sb, "%s/%s", refs_check_dir, iter->relative_path);
-+ ret = fsck_refs_report(o, NULL, sb.buf,
++ ret = fsck_refs_report(o, NULL, sb.buf, NULL,
+ FSCK_MSG_BAD_REF_NAME,
+ "invalid refname format");
+ }
10: 521e0d9ca3 ! 10: 15662c6934 fsck: add ref content check for files backend
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+
+ if (!skip_prefix(pointee_name, "refs/", &p)) {
+
-+ ret = fsck_refs_report(o, NULL, refname,
++ ret = fsck_refs_report(o, NULL, refname, NULL,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
+ "point to target out of refs hierarchy");
+ goto out;
+ }
+
+ if (check_refname_format(pointee_name, 0)) {
-+ ret = fsck_refs_report(o, NULL, refname,
++ ret = fsck_refs_report(o, NULL, refname, NULL,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
+ "point to invalid refname");
+ }
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ goto out;
+
+ if (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {
-+ ret = fsck_refs_report(o, NULL, refname,
++ ret = fsck_refs_report(o, NULL, refname, NULL,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
+ "point to invalid target");
+ goto out;
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+
+ if (!skip_prefix(pointee_path.buf,
+ abs_gitdir.buf, &pointee_name)) {
-+ ret = fsck_refs_report(o, NULL, refname.buf,
++ ret = fsck_refs_report(o, NULL, refname.buf, NULL,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
+ "point to target outside gitdir");
+ goto clean;
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ if (parse_loose_ref_contents(ref_content.buf, &oid,
+ &referent, &type,
+ &failure_errno, &trailing)) {
-+ ret = fsck_refs_report(o, NULL, refname.buf,
++ ret = fsck_refs_report(o, NULL, refname.buf, NULL,
+ FSCK_MSG_BAD_REF_CONTENT,
+ "invalid ref content");
+ goto clean;
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ * be reported as a warning.
+ */
+ if (trailing && (*trailing != '\0' && *trailing != '\n')) {
-+ ret = fsck_refs_report(o, NULL, refname.buf,
++ ret = fsck_refs_report(o, NULL, refname.buf, NULL,
+ FSCK_MSG_TRAILING_REF_CONTENT,
+ "trailing garbage in ref");
+ goto clean;Signature · 1 line
2.45.2
PATCH v11 1/10 fsck: rename "skiplist" to "skip_oids"
The "skiplist" field in "fsck_options" is related to objects. Because we are going to introduce ref consistency check, the "skiplist" name is too general which will make the caller think "skiplist" is related to both the refs and objects.
It may seem that for both refs and objects, we should provide a general "skiplist" here. However, the type for "skiplist" is struct oidset which is totally unsuitable for refs.
To avoid above ambiguity, rename "skiplist" to "skip_oids".
--- fsck.c | 4 ++-- fsck.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
fsck.c +2 -2
@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)if (!strcmp (buf,"skiplist" )) {if (equal == len)die ("skiplist requires a path" ); -oidset_parse_file (&options->skiplist, buf + equal +1 , +oidset_parse_file (&options->skip_oids, buf + equal +1 , the_repository->hash_algo); buf += len +1 ;continue ;
@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)static int object_on_skiplist (struct fsck_options *opts,const struct object_id *oid) { -return opts && oid &&oidset_contains (&opts->skiplist, oid); +return opts && oid &&oidset_contains (&opts->skip_oids, oid); } __attribute__((format (printf, 5, 6)))
fsck.h +2 -2
@@ -136,7 +136,7 @@ struct fsck_options {fsck_error error_func; unsigned strict:1;enum fsck_msg_type *msg_type; -struct oidset skiplist; +struct oidset skip_oids;struct oidset gitmodules_found;struct oidset gitmodules_done;struct oidset gitattributes_found;
@@ -145,7 +145,7 @@ struct fsck_options { };#define FSCK_OPTIONS_DEFAULT { \ - .skiplist = OIDSET_INIT, \ + .skip_oids = OIDSET_INIT, \ .gitmodules_found = OIDSET_INIT, \ .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \
Signature · 1 line
2.45.2
PATCH v11 2/10 fsck: add a unified interface for reporting fsck messages
In reply to shejialuoThe static function "report" provided by "fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "fsck_vreport" following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.
When checking loose refs and reflogs, we only need to pass the checked name to the fsck error report function. However, for packed-refs and reftable refs, we need to check both the consistency of the file itself and the refs or reflogs contained in the file. In order to provide above checks, add two parameters "ref_checkee" and "sub_ref_checkee" in "fsck_vreport" function.
Like "report", the "fsck_vreport" function will use "error_func" registered in "fsck_options" to report customized messages. Change "error_func" prototype to align with the new "fsck_vreport".
Then, change "report" function to use "fsck_vreport" to report objects related messages. Add a new function called "fsck_refs_report" to use "fsck_vreport" to report refs related messages.
--- builtin/fsck.c | 2 ++ builtin/mktag.c | 2 ++ fsck.c | 60 ++++++++++++++++++++++++++++++++++++++++++------- fsck.h | 19 +++++++++++++++- object-file.c | 12 +++++----- 5 files changed, 81 insertions(+), 14 deletions(-)
builtin/fsck.c +2 -0
@@ -92,6 +92,8 @@ static int objerror(struct object *obj, const char *err) static int fsck_error_func(struct fsck_options *o UNUSED,const struct object_id *oid,enum object_type object_type, + const char *ref_checkee UNUSED, + const char *sub_ref_checkee UNUSED,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message)
builtin/mktag.c +2 -0
@@ -20,6 +20,8 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT; static int mktag_fsck_error_func(struct fsck_options *o UNUSED, const struct object_id *oid UNUSED, enum object_type object_type UNUSED, + const char *ref_checkee UNUSED, + const char *sub_ref_checkee UNUSED,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message)
fsck.c +52 -8
96 lines not on this page · raw message
fsck.h +18 -1
@@ -114,19 +114,25 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);typedef int (*fsck_walk_func )(struct object *obj,enum object_type object_type,void *data,struct fsck_options *options); -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */ +/* +* callback function for reporting errors when checking either objects or refs +*/ typedef int (*fsck_error )(struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const char *ref_checkee,const char *sub_ref_checkee,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);int fsck_error_function (struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const char *ref_checkee,const char *sub_ref_checkee,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const char *ref_checkee, +const char *sub_ref_checkee,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);
@@ -209,6 +215,17 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +/* +* Report an error or warning for refs. +*/ +__attribute__((format (printf,6 ,7 ))) +int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +const char *ref_checkee, +const char *sub_ref_checkee, +enum fsck_msg_id msg_id, +const char *fmt, ...); +/* * Subsystem for storing human-readable names for each object. *
object-file.c +7 -5
@@ -2470,11 +2470,13 @@ int repo_has_object_file(struct repository *r,* give more context. */ static int hash_format_check_report(struct fsck_options *opts UNUSED, - const struct object_id *oid UNUSED, - enum object_type object_type UNUSED, - enum fsck_msg_type msg_type UNUSED, - enum fsck_msg_id msg_id UNUSED, -const char *message) + const struct object_id *oid UNUSED, + enum object_type object_type UNUSED, + const char *ref_checkee UNUSED, + const char *sub_ref_checkee UNUSED, + enum fsck_msg_type msg_type UNUSED, + enum fsck_msg_id msg_id UNUSED, +const char *message) {error (_ ("object fails fsck: %s" ), message);return 1 ;
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v11 02/10] fsck: add a unified interface for reporting fsck messages
Quoted 8 lines · shejialuo
The static function "report" provided by "
fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "fsck_vreport" following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.
Quoted 6 lines
When checking loose refs and reflogs, we only need to pass the checked name to the fsck error report function. However, for packed-refs and reftable refs, we need to check both the consistency of the file itself and the refs or reflogs contained in the file. In order to provide above checks, add two parameters "
ref_checkee" and "sub_ref_checkee" in "fsck_vreport" function.
Nit: It would be nice, if you described here, what is the expected usage of "ref_checkee" and "sub_ref_checkee".
[snip]
Quoted 5 lines
fsck.h +0
-0
fsck.h int is_valid_msg_type(const char *msg_id, const char *msg_type); · line 122
Quoted 10 lines
fsck.h +4
-1@@ -114,19 +114,25 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);typedef int (*fsck_walk_func )(struct object *obj,enum object_type object_type,void *data,struct fsck_options *options); -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */ +/* +* callback function for reporting errors when checking either objects or refs +*/ typedef int (*fsck_error)(struct fsck_options *o, const struct object_id *oid, enum object_type object_type, + const char *ref_checkee, const char *sub_ref_checkee,
This makes me really wonder if this is the best way we can do this? This seems to solve for the current situation, but what happens if you want to also adding the reftable size or packed-refs size here? Would you introduce another field?
would it be better to add a single const struct *fsck_refs_info instead?
Perhaps something like:
struct fsck_refs_info {
char *refname;
union {
struct {
...
} reftable;
struct {
...
} files;
} u;
}Of course we can fill in the details as we need them.
Quoted 59 lines
fsck.h +14
-0enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message);int fsck_error_function (struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const char *ref_checkee,const char *sub_ref_checkee,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const char *ref_checkee, +const char *sub_ref_checkee,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);@@ -209,6 +215,17 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +/* +* Report an error or warning for refs. +*/ +__attribute__((format (printf,6 ,7 ))) +int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +const char *ref_checkee, +const char *sub_ref_checkee, +enum fsck_msg_id msg_id, +const char *fmt, ...); +/* * Subsystem for storing human-readable names for each object. * object-file.c +7
-6@@ -2470,11 +2470,13 @@ int repo_has_object_file(struct repository *r,* give more context. */ static int hash_format_check_report(struct fsck_options *opts UNUSED, - const struct object_id *oid UNUSED, - enum object_type object_type UNUSED, - enum fsck_msg_type msg_type UNUSED, - enum fsck_msg_id msg_id UNUSED, -const char *message) + const struct object_id *oid UNUSED, + enum object_type object_type UNUSED, + const char *ref_checkee UNUSED, + const char *sub_ref_checkee UNUSED, + enum fsck_msg_type msg_type UNUSED, + enum fsck_msg_id msg_id UNUSED, +const char *message) {error (_ ("object fails fsck: %s" ), message);return 1 ; --2.45.2
Re: [GSoC][PATCH v11 02/10] fsck: add a unified interface for reporting fsck messages
Quoted 40 lines · Karthik Nayak
Quoted 5 lines · shejialuo
fsck.h +0
-0fsck.h int is_valid_msg_type(const char *msg_id, const char *msg_type); · line 122
Quoted 10 lines
fsck.h +4
-1@@ -114,19 +114,25 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);typedef int (*fsck_walk_func )(struct object *obj,enum object_type object_type,void *data,struct fsck_options *options); -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */ +/* +* callback function for reporting errors when checking either objects or refs +*/ typedef int (*fsck_error)(struct fsck_options *o, const struct object_id *oid, enum object_type object_type, + const char *ref_checkee, const char *sub_ref_checkee,This makes me really wonder if this is the best way we can do this? This seems to solve for the current situation, but what happens if you want to also adding the reftable size or packed-refs size here? Would you introduce another field?
would it be better to add a single
const struct *fsck_refs_infoinstead?Perhaps something like:
struct fsck_refs_info { char *refname; union { struct { ... } reftable; struct { ... } files; } u; }
Quoted 2 lines
Of course we can fill in the details as we need them.
I agree, we should design an extensible data structure here. I will use this idea. Because we don't know what we will do in the current time. However, I think "refname" is not good, instead I decide to use "ref_checkee", "refname" may let caller think we only check the refname. However, we need to also check reflog.
PATCH v11 3/10 fsck: rename objects-related fsck error functions
In reply to shejialuoThe names of objects-related fsck error functions are general. It's OK when there is only object database check. However, we have introduced refs database check report function. To avoid ambiguity, rename object-related fsck error functions to explicitly indicate these functions are used to report objects-related messages.
--- builtin/fsck.c | 18 +++++++++--------- fsck.c | 37 +++++++++++++++++++------------------ fsck.h | 32 ++++++++++++++++---------------- 3 files changed, 44 insertions(+), 43 deletions(-)
builtin/fsck.c +9 -9
@@ -89,14 +89,14 @@ static int objerror(struct object *obj, const char *err)return -1 ; } -static int fsck_error_func(struct fsck_options *o UNUSED, -const struct object_id *oid, -enum object_type object_type, - const char *ref_checkee UNUSED, - const char *sub_ref_checkee UNUSED, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +static int fsck_objects_error_func(struct fsck_options *o UNUSED, +const struct object_id *oid, +enum object_type object_type, + const char *ref_checkee UNUSED, + const char *sub_ref_checkee UNUSED, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {switch (msg_type) {case FSCK_WARN :
@@ -940,7 +940,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) fsck_walk_options.walk = mark_object; fsck_obj_options.walk = mark_used; - fsck_obj_options.error_func = fsck_error_func; + fsck_obj_options.error_func = fsck_objects_error_func;if (check_strict) fsck_obj_options.strict =1 ;
fsck.c +19 -18
@@ -1239,14 +1239,14 @@ int fsck_buffer(const struct object_id *oid, enum object_type type, type); } -int fsck_error_function (struct fsck_options *o, -const struct object_id *oid, - enum object_type object_type UNUSED, - const char *ref_checkee UNUSED, - const char *sub_ref_checkee UNUSED, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +int fsck_objects_error_function (struct fsck_options *o, +const struct object_id *oid, + enum object_type object_type UNUSED, + const char *ref_checkee UNUSED, + const char *sub_ref_checkee UNUSED, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {if (msg_type ==FSCK_WARN ) {warning ("object %s: %s" ,fsck_describe_object (o, oid), message);
@@ -1344,19 +1344,20 @@ int git_fsck_config(const char *var, const char *value,* Custom error callbacks that are used in more than one place. */ -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, -const char *ref_checkee, -const char *sub_ref_checkee, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message) +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +const struct object_id *oid, +enum object_type object_type, +const char *ref_checkee, +const char *sub_ref_checkee, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message) {if (msg_id ==FSCK_MSG_GITMODULES_MISSING ) {puts (oid_to_hex (oid));return 0 ; } -return fsck_error_function (o, oid, object_type, ref_checkee, - sub_ref_checkee, msg_type, msg_id, message); +return fsck_objects_error_function (o, oid, object_type, ref_checkee, + sub_ref_checkee, msg_type, msg_id, + message); }
fsck.h +16 -16
@@ -123,19 +123,19 @@ typedef int (*fsck_error)(struct fsck_options *o, enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); -int fsck_error_function (struct fsck_options *o, -const struct object_id *oid,enum object_type object_type, -const char *ref_checkee,const char *sub_ref_checkee, -enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, -const char *message); -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, -const char *ref_checkee, -const char *sub_ref_checkee, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message); +int fsck_objects_error_function (struct fsck_options *o, +const struct object_id *oid,enum object_type object_type, +const char *ref_checkee,const char *sub_ref_checkee, +enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, +const char *message); +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +const struct object_id *oid, +enum object_type object_type, +const char *ref_checkee, +const char *sub_ref_checkee, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);struct fsck_options {fsck_walk_func walk;
@@ -156,7 +156,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function \ + .error_func = fsck_objects_error_function \ }#define FSCK_OPTIONS_STRICT { \ .strict = 1, \
@@ -164,7 +164,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function, \ + .error_func = fsck_objects_error_function, \ }#define FSCK_OPTIONS_MISSING_GITMODULES { \ .strict = 1, \
@@ -172,7 +172,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_cb_print_missing_gitmodules, \ + .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ }/* descend in all linked child objects
Signature · 1 line
2.45.2
PATCH v11 4/10 fsck: add refs-related error report function
In reply to shejialuoCreate refs-specific "error_func" callback "fsck_refs_error_function" which could provide the following report messages.
1. "ref_checkee": "fsck error name": "user message".
2. "ref_checkee.sub_ref_checkee": "fsck error name": "user message".
3. "ref_checkee -> (oid hex)": "fsck error name": "user message".
4. "ref_checkee.sub_ref_checkee -> (oid hex)": "fsck error name": "user message".
"fsck_refs_error_function" uses the "ref_checkee" and "sub_ref_checkee" parameters to indicate the information of the checked refs. For loose ref and reflog, it only uses the "ref_checkee" parameter. For packed refs and reftable refs, when checking the consistency of the file itself, it still only uses "ref_checkee" parameter. However, when checking the consistency of the ref or reflog contained in the file, it will use "sub_ref_checkee" parameter to indicate that we are not checking the file but the incorporated ref or reflog.
"fsck_refs_error_function" will use the "oid" parameter if the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.
Last, add "FSCK_REFS_OPTIONS_DEFAULT" and "FSCK_REFS_OPTIONS_STRICT" macros to create refs options easily.
--- fsck.c | 29 +++++++++++++++++++++++++++++ fsck.h | 15 +++++++++++++++ 2 files changed, 44 insertions(+)
fsck.c +29 -0
@@ -1256,6 +1256,35 @@ int fsck_objects_error_function(struct fsck_options *o,return 1 ; } +int fsck_refs_error_function(struct fsck_options *options UNUSED, +const struct object_id *oid, + enum object_type object_type UNUSED, +const char *ref_checkee, +const char *sub_ref_checkee, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) +{ +struct strbuf sb =STRBUF_INIT ; +int ret =0 ; + +if (sub_ref_checkee) +strbuf_addf (&sb,"%s.%s" , ref_checkee, sub_ref_checkee); +else +strbuf_addstr (&sb, ref_checkee); + +if (oid) +strbuf_addf (&sb," -> (%s)" ,oid_to_hex (oid)); + +if (msg_type ==FSCK_WARN ) +warning ("%s: %s" , sb.buf, message); +else + ret =error ("%s: %s" , sb.buf, message); + +strbuf_release (&sb); +return ret; +} +static int fsck_blobs (struct oidset *blobs_found,struct oidset *blobs_done,enum fsck_msg_id msg_missing,enum fsck_msg_id msg_type,struct fsck_options *options,const char *blob_type)
fsck.h +15 -0
@@ -136,6 +136,14 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); +int fsck_refs_error_function (struct fsck_options *options, +const struct object_id *oid, +enum object_type object_type, +const char *ref_checkee, +const char *sub_ref_checkee, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);struct fsck_options {fsck_walk_func walk;
@@ -174,6 +182,13 @@ struct fsck_options { .gitattributes_done = OIDSET_INIT, \ .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ } +#define FSCK_REFS_OPTIONS_DEFAULT { \ + .error_func = fsck_refs_error_function, \ +} +#define FSCK_REFS_OPTIONS_STRICT { \ + .strict = 1, \ + .error_func = fsck_refs_error_function, \ +}/* descend in all linked child objects * the return value is:
Signature · 1 line
2.45.2
PATCH v11 5/10 refs: set up ref consistency check infrastructure
In reply to shejialuoThe interfaces defined in the ref_storage_be are carefully structured in semantic. It's organized as the five parts:
1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.
To keep consistent with the git-fsck(1), add a new interface named "fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.
Last, implement placeholder functions for each ref backends.
--- refs.c | 5 +++++ refs.h | 8 ++++++++ refs/debug.c | 11 +++++++++++ refs/files-backend.c | 15 ++++++++++++++- refs/packed-backend.c | 8 ++++++++ refs/refs-internal.h | 6 ++++++ refs/reftable-backend.c | 8 ++++++++ 7 files changed, 60 insertions(+), 1 deletion(-)
refs.c +5 -0
@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)return check_or_sanitize_refname (refname, flags,NULL ); } +int refs_fsck (struct ref_store *refs,struct fsck_options *o) +{ +return refs->be->fsck (refs, o); +} +void sanitize_refname_component (const char *refname,struct strbuf *out) {if (check_or_sanitize_refname (refname,REFNAME_ALLOW_ONELEVEL , out))
refs.h +8 -0
@@ -4,6 +4,7 @@#include "commit.h" #include "repository.h" +struct fsck_options ;struct object_id ;struct ref_store ;struct strbuf ;
@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat*/ int check_refname_format (const char *refname,int flags); +/* +* Check the reference database for consistency. Return 0 if refs and +* reflogs are consistent, and non-zero otherwise. The errors will be +* written to stderr. +*/ +int refs_fsck (struct ref_store *refs,struct fsck_options *o); +/* * Apply the rules from check_refname_format, but mutate the result until it * is acceptable, and place the result in "out".
refs/debug.c +11 -0
@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,return res; } +static int debug_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store; +int res = drefs->refs->be->fsck (drefs->refs, o); +trace_printf_key (&trace_refs,"fsck: %d\n" , res); +return res; +} +struct ref_storage_be refs_be_debug = { .name ="debug" , .init =NULL ,
@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = { .create_reflog = debug_create_reflog, .delete_reflog = debug_delete_reflog, .reflog_expire = debug_reflog_expire, + + .fsck = debug_fsck, };
refs/files-backend.c +14 -1
@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +static int files_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +int ret; +struct files_ref_store *refs = +files_downcast (ref_store,REF_STORE_READ ,"fsck" ); + + ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); +return ret; +} +struct ref_storage_be refs_be_files = { .name ="files" , .init = files_ref_store_init,
@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = { .reflog_exists = files_reflog_exists, .create_reflog = files_create_reflog, .delete_reflog = files_delete_reflog, - .reflog_expire = files_reflog_expire + .reflog_expire = files_reflog_expire, + + .fsck = files_fsck, };
refs/packed-backend.c +8 -0
@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_sreturn empty_ref_iterator_begin (); } +static int packed_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_packed = { .name ="packed" , .init = packed_ref_store_init,
@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = { .create_reflog = NULL, .delete_reflog = NULL, .reflog_expire = NULL, + + .fsck = packed_fsck, };
refs/refs-internal.h +6 -0
@@ -4,6 +4,7 @@#include "refs.h" #include "iterator.h" +struct fsck_options ;struct ref_transaction ;/*
@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,typedef int read_symbolic_ref_fn (struct ref_store *ref_store,const char *refname,struct strbuf *referent); +typedef int fsck_fn (struct ref_store *ref_store, +struct fsck_options *o); +struct ref_storage_be {const char *name;ref_store_init_fn *init;
@@ -677,6 +681,8 @@ struct ref_storage_be {create_reflog_fn *create_reflog;delete_reflog_fn *delete_reflog;reflog_expire_fn *reflog_expire; + +fsck_fn *fsck; };extern struct ref_storage_be refs_be_files;
refs/reftable-backend.c +8 -0
@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,return ret; } +static int reftable_be_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_reftable = { .name ="reftable" , .init = reftable_be_init,
@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = { .create_reflog = reftable_be_create_reflog, .delete_reflog = reftable_be_delete_reflog, .reflog_expire = reftable_be_reflog_expire, + + .fsck = reftable_be_fsck, };
Signature · 1 line
2.45.2
PATCH v11 6/10 git refs: add verify subcommand
In reply to shejialuoIntroduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency and also this subcommand will be used as the entry point of checking refs for "git-fsck(1)". Last, add "verbose" field into "fsck_options" to indicate whether we should print verbose messages when checking refs and objects consistency.
--- Documentation/git-refs.txt | 13 +++++++++++ builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++ fsck.h | 1 + 3 files changed, 58 insertions(+)
Documentation/git-refs.txt +13 -0
@@ -10,6 +10,7 @@ SYNOPSIS -------- [verse] 'git refs migrate' --ref-format=<format> [--dry-run] +'git refs verify' [--strict] [--verbose] DESCRIPTION -----------
@@ -22,6 +23,9 @@ COMMANDS migrate:: Migrate ref store between different formats. +verify:: + Verify reference database consistency. + OPTIONS -------
@@ -39,6 +43,15 @@ include::ref-storage-format.txt[] can be used to double check that the migration works as expected before performing the actual migration. +The following options are specific to 'git refs verify': + +--strict:: + Enable more strict checking, every WARN severity for the `Fsck Messages` + be seen as ERROR. See linkgit:git-fsck[1]. + +--verbose:: + When verifying the reference database consistency, be chatty. + KNOWN LIMITATIONS -----------------
builtin/refs.c +44 -0
69 lines not on this page · raw message
fsck.h +1 -0
@@ -149,6 +149,7 @@ struct fsck_options {fsck_walk_func walk;fsck_error error_func; unsigned strict:1; + unsigned verbose:1;enum fsck_msg_type *msg_type;struct oidset skip_oids;struct oidset gitmodules_found;
Signature · 1 line
2.45.2
PATCH v11 7/10 builtin/fsck: add `git-refs verify` child process
In reply to shejialuoIntroduce a new function "fsck_refs" that initializes and runs a child process to execute the "git-refs verify" command.
--- builtin/fsck.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
builtin/fsck.c +17 -0
@@ -898,6 +898,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)return res; } +static void fsck_refs (void ) +{ +struct child_process refs_verify =CHILD_PROCESS_INIT ; +child_process_init (&refs_verify); + refs_verify.git_cmd =1 ; +strvec_pushl (&refs_verify.args,"refs" ,"verify" ,NULL ); +if (verbose) +strvec_push (&refs_verify.args,"--verbose" ); +if (check_strict) +strvec_push (&refs_verify.args,"--strict" ); + +if (run_command (&refs_verify)) + errors_found |=ERROR_REFS ; +} +static char const *const fsck_usage[] = { N_("git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\n" " [--[no-]full] [--strict] [--verbose] [--lost-found]\n"
@@ -1067,6 +1082,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)check_connectivity (); +fsck_refs (); +if (the_repository->settings.core_commit_graph) {struct child_process commit_graph_verify =CHILD_PROCESS_INIT ;
Signature · 1 line
2.45.2
PATCH v11 8/10 files-backend: add unified interface for refs scanning
In reply to shejialuoFor refs and reflogs, we need to scan its corresponding directories to check every regular file or symbolic link which shares the same pattern. Introduce a unified interface for scanning directories for files-backend.
--- refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 76 insertions(+), 1 deletion(-)
refs/files-backend.c +76 -1
@@ -6,6 +6,7 @@#include "../gettext.h" #include "../hash.h" #include "../hex.h" +#include "../fsck.h" #include "../refs.h" #include "refs-internal.h" #include "ref-cache.h"
@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +/* +* For refs and reflogs, they share a unified interface when scanning +* the whole directory. This function is used as the callback for each +* regular file or symlink in the directory. +*/ +typedef int (*files_fsck_refs_fn )(struct fsck_options *o, +const char *gitdir, +const char *refs_check_dir, +struct dir_iterator *iter); + +static int files_fsck_refs_dir (struct ref_store *ref_store, +struct fsck_options *o, +const char *refs_check_dir, +files_fsck_refs_fn *fsck_refs_fns) +{ +const char *gitdir = ref_store->gitdir; +struct strbuf sb =STRBUF_INIT ; +struct dir_iterator *iter; +int iter_status; +int ret =0 ; + +strbuf_addf (&sb,"%s/%s" , gitdir, refs_check_dir); + + iter =dir_iterator_begin (sb.buf,0 ); + +if (!iter) { + ret =error_errno ("cannot open directory %s" , sb.buf); + goto out; + } + +while ((iter_status =dir_iterator_advance (iter)) ==ITER_OK ) { +if (S_ISDIR (iter->st.st_mode)) { +continue ; + }else if (S_ISREG (iter->st.st_mode) || +S_ISLNK (iter->st.st_mode)) { +if (o->verbose) +fprintf_ln (stderr,"Checking %s/%s" , + refs_check_dir, iter->relative_path); +for (size_t i =0 ; fsck_refs_fns[i]; i++) { +if (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter)) + ret =-1 ; + } + }else { + ret =error (_ ("unexpected file type for '%s'" ), + iter->basename); + } + } + +if (iter_status !=ITER_DONE ) + ret =error (_ ("failed to iterate over '%s'" ), sb.buf); + +out: +strbuf_release (&sb); +return ret; +} + +static int files_fsck_refs (struct ref_store *ref_store, +struct fsck_options *o) +{ +int ret; +files_fsck_refs_fn fsck_refs_fns[]= { +NULL + }; + +if (o->verbose) +fprintf_ln (stderr,"Checking references consistency" ); + + ret =files_fsck_refs_dir (ref_store, o,"refs" , fsck_refs_fns); + +return ret; +} +static int files_fsck (struct ref_store *ref_store,struct fsck_options *o) {
@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,struct files_ref_store *refs =files_downcast (ref_store,REF_STORE_READ ,"fsck" ); - ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); + ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o) + |files_fsck_refs (ref_store, o); +return ret; }
Signature · 1 line
2.45.2
PATCH v11 9/10 fsck: add ref name check for files backend
In reply to shejialuoThe git-fsck(1) only implicitly checks the reference, it does not fully check refs with bad format name such as standalone "@" and name ending with ".lock".
In order to provide such checks, add a new fsck message id "badRefName" with default ERROR type. Use existing "check_refname_format" to explicit check the ref name. And add a new unit test to verify the functionality.
--- Documentation/fsck-msgids.txt | 3 + fsck.h | 1 + refs/files-backend.c | 20 +++++++ t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++ 4 files changed, 125 insertions(+) create mode 100755 t/t0602-reffiles-fsck.sh
Documentation/fsck-msgids.txt +3 -0
@@ -19,6 +19,9 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefName`:: + (ERROR) A ref has a bad name. + `badTagName`:: (INFO) A tag has an invalid format.
fsck.h +1 -0
@@ -31,6 +31,7 @@ enum fsck_msg_type { FUNC(BAD_NAME, ERROR) \ FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ + FUNC(BAD_REF_NAME, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
refs/files-backend.c +20 -0
@@ -3419,6 +3419,25 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,const char *refs_check_dir, struct dir_iterator *iter); +static int files_fsck_refs_name (struct fsck_options *o, + const char *gitdir UNUSED, +const char *refs_check_dir, +struct dir_iterator *iter) +{ +struct strbuf sb =STRBUF_INIT ; +int ret =0 ; + +if (check_refname_format (iter->basename,REFNAME_ALLOW_ONELEVEL )) { +strbuf_addf (&sb,"%s/%s" , refs_check_dir, iter->relative_path); + ret =fsck_refs_report (o,NULL , sb.buf,NULL , +FSCK_MSG_BAD_REF_NAME , +"invalid refname format" ); + } + +strbuf_release (&sb); +return ret; +} + static int files_fsck_refs_dir(struct ref_store *ref_store, struct fsck_options *o, const char *refs_check_dir,
@@ -3470,6 +3489,7 @@ static int files_fsck_refs(struct ref_store *ref_store, {int ret;files_fsck_refs_fn fsck_refs_fns[]= { + files_fsck_refs_name,NULL };
t/t0602-reffiles-fsck.sh new file +101 -0
101 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v11 10/10 fsck: add ref content check for files backend
In reply to shejialuoEnhance the git-fsck(1) command by adding a check for reference content in the files backend. The new functionality ensures that symrefs, real symbolic link and regular refs are validated correctly.
In order to check the trailing content of the regular refs, add a new parameter trailing to parse_loose_ref_contents.
For symrefs, parse_loose_ref_contents will set the "referent". However, symbolic link could be either absolute or relative. Use "strbuf_add_real_path" to read the symbolic link and convert the relative path to absolute path. Then use "skip_prefix" to make it align with symref "referent".
Thus, the symrefs and symbolic links could share the same interface. Add a new function "files_fsck_symref_target" which aims at checking the following things:
1. whether the pointee is under the refs/ directory.
2. whether the pointee name is correct.
3. whether the pointee path is a wrong type in filesystem.
Last, add the following FSCK MESSAGEs:
1. "badRefContent(ERROR)": A ref has a bad content
2. "badSymrefPointee(ERROR)": The pointee of a symref is bad.
3. "trailingRefContent(WARN)": A ref content has trailing contents.
--- Documentation/fsck-msgids.txt | 9 +++ fsck.h | 3 + refs.c | 2 +- refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++- refs/refs-internal.h | 5 +- t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++ 6 files changed, 269 insertions(+), 5 deletions(-)
Documentation/fsck-msgids.txt +9 -0
@@ -19,9 +19,15 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefContent`:: + (ERROR) A ref has a bad content. + `badRefName`:: (ERROR) A ref has a bad name. +`badSymrefPointee`:: + (ERROR) The pointee of a symref is bad. + `badTagName`:: (INFO) A tag has an invalid format.
@@ -167,6 +173,9 @@ `nullSha1`:: (WARN) Tree contains entries pointing to a null sha1. +`trailingRefContent`:: + (WARN) A ref content has trailing contents. + `treeNotSorted`:: (ERROR) A tree is not properly sorted.
fsck.h +3 -0
@@ -32,6 +32,8 @@ enum fsck_msg_type { FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ FUNC(BAD_REF_NAME, ERROR) \ + FUNC(BAD_REF_CONTENT, ERROR) \ + FUNC(BAD_SYMREF_POINTEE, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
@@ -72,6 +74,7 @@ enum fsck_msg_type { FUNC(HAS_DOTDOT, WARN) \ FUNC(HAS_DOTGIT, WARN) \ FUNC(NULL_SHA1, WARN) \ + FUNC(TRAILING_REF_CONTENT, WARN) \ FUNC(ZERO_PADDED_FILEMODE, WARN) \ FUNC(NUL_IN_COMMIT, WARN) \FUNC (LARGE_PATHNAME ,WARN ) \
refs.c +1 -1
@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store, } result =parse_loose_ref_contents (content.buf, oid, referent, type, - failure_errno); + failure_errno,NULL ); done:strbuf_release (&full_path);
refs/files-backend.c +143 -2
181 lines not on this page · raw message
refs/refs-internal.h +3 -2
@@ -709,11 +709,12 @@ struct ref_store {/* * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for -* invalid contents. +* invalid contents. Also *trailing is set to the first character after the +* refname or NULL if the referent is not empty. */ int parse_loose_ref_contents (const char *buf,struct object_id *oid,struct strbuf *referent,unsigned int *type, -int *failure_errno); +int *failure_errno,const char **trailing);/* * Fill in the generic part of refs and add it to our collection of
t/t0602-reffiles-fsck.sh +110 -0
114 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend
Quoted 256 lines · shejialuo
Enhance the git-fsck(1) command by adding a check for reference content in the files backend. The new functionality ensures that symrefs, real symbolic link and regular refs are validated correctly.
In order to check the trailing content of the regular refs, add a new parameter
trailingtoparse_loose_ref_contents.For symrefs,
parse_loose_ref_contentswill set the "referent". However, symbolic link could be either absolute or relative. Use "strbuf_add_real_path" to read the symbolic link and convert the relative path to absolute path. Then use "skip_prefix" to make it align with symref "referent".Thus, the symrefs and symbolic links could share the same interface. Add a new function "
files_fsck_symref_target" which aims at checking the following things:1. whether the pointee is under the
refs/directory.
2. whether the pointee name is correct.
3. whether the pointee path is a wrong type in filesystem.Last, add the following FSCK MESSAGEs:
1. "badRefContent(ERROR)": A ref has a bad content
2. "badSymrefPointee(ERROR)": The pointee of a symref is bad.
3. "trailingRefContent(WARN)": A ref content has trailing contents.Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- Documentation/fsck-msgids.txt | 9 +++ fsck.h | 3 + refs.c | 2 +- refs/files-backend.c | 145 +++++++++++++++++++++++++++++++++- refs/refs-internal.h | 5 +- t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++ 6 files changed, 269 insertions(+), 5 deletions(-)Documentation/fsck-msgids.txt +9
-0@@ -19,9 +19,15 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefContent`:: + (ERROR) A ref has a bad content. + `badRefName`:: (ERROR) A ref has a bad name. +`badSymrefPointee`:: + (ERROR) The pointee of a symref is bad. + `badTagName`:: (INFO) A tag has an invalid format.@@ -167,6 +173,9 @@ `nullSha1`:: (WARN) Tree contains entries pointing to a null sha1. +`trailingRefContent`:: + (WARN) A ref content has trailing contents. + `treeNotSorted`:: (ERROR) A tree is not properly sorted.fsck.h +3
-0@@ -32,6 +32,8 @@ enum fsck_msg_type { FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ FUNC(BAD_REF_NAME, ERROR) \ + FUNC(BAD_REF_CONTENT, ERROR) \ + FUNC(BAD_SYMREF_POINTEE, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \@@ -72,6 +74,7 @@ enum fsck_msg_type { FUNC(HAS_DOTDOT, WARN) \ FUNC(HAS_DOTGIT, WARN) \ FUNC(NULL_SHA1, WARN) \ + FUNC(TRAILING_REF_CONTENT, WARN) \ FUNC(ZERO_PADDED_FILEMODE, WARN) \ FUNC(NUL_IN_COMMIT, WARN) \FUNC (LARGE_PATHNAME ,WARN ) \refs.c +1
-1@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store, } result =parse_loose_ref_contents (content.buf, oid, referent, type, - failure_errno); + failure_errno,NULL ); done:strbuf_release (&full_path);refs/files-backend.c +115
-2@@ -1,6 +1,7 @@#define USE_THE_REPOSITORY_VARIABLE #include "../git-compat-util.h" +#include "../abspath.h" #include "../copy.h" #include "../environment.h" #include "../gettext.h" @@ -553,7 +554,7 @@ static int read_ref_internal(struct ref_store *ref_store, const char *refname,strbuf_rtrim (&sb_contents); buf = sb_contents.buf; - ret =parse_loose_ref_contents (buf, oid, referent, type, &myerr); + ret =parse_loose_ref_contents (buf, oid, referent, type, &myerr,NULL ); out:if (ret && !myerr)@@ -589,7 +590,7 @@ static int files_read_symbolic_ref(struct ref_store *ref_store, const char *refnint parse_loose_ref_contents (const char *buf,struct object_id *oid,struct strbuf *referent,unsigned int *type, -int *failure_errno) +int *failure_errno,const char **trailing) {const char *p;if (skip_prefix (buf,"ref:" , &buf)) {@@ -611,6 +612,10 @@ int parse_loose_ref_contents(const char *buf, struct object_id *oid, *failure_errno =EINVAL ;return -1 ; } + +if (trailing) + *trailing = p; +return 0 ; }@@ -3438,6 +3443,141 @@ static int files_fsck_refs_name(struct fsck_options *o,return ret; } +/* +* Check the symref "pointee_name" and "pointee_path". The caller should +* make sure that "pointee_path" is absolute. For symbolic ref, "pointee_name" +* would be the content after "refs:". For symblic link, "pointee_name" would +* be the relative path agaignst "gitdir". +*/ +static int files_fsck_symref_target (struct fsck_options *o, +const char *refname, +const char *pointee_name, +const char *pointee_path) +{ +const char *p =NULL ; +struct stat st; +int ret =0 ; + +if (!skip_prefix (pointee_name,"refs/" , &p)) { + + ret =fsck_refs_report (o,NULL , refname,NULL , +FSCK_MSG_BAD_SYMREF_POINTEE , +"point to target out of refs hierarchy" ); + goto out; + } + +if (check_refname_format (pointee_name,0 )) { + ret =fsck_refs_report (o,NULL , refname,NULL , +FSCK_MSG_BAD_SYMREF_POINTEE , +"point to invalid refname" ); + } + +if (lstat (pointee_path, &st) <0 ) + goto out; + +if (!S_ISREG (st.st_mode) && !S_ISLNK (st.st_mode)) { + ret =fsck_refs_report (o,NULL , refname,NULL , +FSCK_MSG_BAD_SYMREF_POINTEE , +"point to invalid target" ); + goto out; + } +out: +return ret; +} + +static int files_fsck_refs_content (struct fsck_options *o, +const char *gitdir, +const char *refs_check_dir, +struct dir_iterator *iter) +{ +struct strbuf pointee_path =STRBUF_INIT , + ref_content =STRBUF_INIT , + abs_gitdir =STRBUF_INIT , + referent =STRBUF_INIT , + refname =STRBUF_INIT ; +const char *trailing =NULL ; +int failure_errno =0 ; +unsigned int type =0 ; +struct object_id oid; +int ret =0 ; + +strbuf_addf (&refname,"%s/%s" , refs_check_dir, iter->relative_path); + +/* +* If the file is a symlink, we need to only check the connectivity +* of the destination object. +*/ +if (S_ISLNK (iter->st.st_mode)) { +const char *pointee_name =NULL ; + +strbuf_add_real_path (&pointee_path, iter->path.buf); + +strbuf_add_absolute_path (&abs_gitdir, gitdir); +strbuf_normalize_path (&abs_gitdir); +if (!is_dir_sep (abs_gitdir.buf[abs_gitdir.len -1 ])) +strbuf_addch (&abs_gitdir,'/' ); + +if (!skip_prefix (pointee_path.buf, + abs_gitdir.buf, &pointee_name)) { + ret =fsck_refs_report (o,NULL , refname.buf,NULL , +FSCK_MSG_BAD_SYMREF_POINTEE , +"point to target outside gitdir" ); + goto clean; + } + + ret =files_fsck_symref_target (o, refname.buf, pointee_name, + pointee_path.buf); + goto clean; + } + +if (strbuf_read_file (&ref_content, iter->path.buf,0 ) <0 ) { + ret =error_errno (_ ("%s/%s: unable to read the ref" ), + refs_check_dir, iter->relative_path); + goto clean; + } + +if (parse_loose_ref_contents (ref_content.buf, &oid, + &referent, &type, + &failure_errno, &trailing)) { + ret =fsck_refs_report (o,NULL , refname.buf,NULL , +FSCK_MSG_BAD_REF_CONTENT , +"invalid ref content" ); + goto clean; + } + +/* +* If the ref is a symref, we need to check the destination name and +* connectivity. +*/ +if (referent.len && (type &REF_ISSYMREF )) { +strbuf_addf (&pointee_path,"%s/%s" , gitdir, referent.buf);
refs/files-backend.c static int files_fsck_refs_name(struct fsck_options *o, · line 3563
Quoted 10 lines
refs/files-backend.c +10
-0+strbuf_rtrim (&referent); + + ret =files_fsck_symref_target (o, refname.buf, referent.buf, + pointee_path.buf); + goto clean; + }else { +/* +* Only regular refs could have a trailing garbage. Should +* be reported as a warning. +*/
What happens if a symbolic reference has trailing garbage ?
Quoted 168 lines
refs/files-backend.c +18
-0+if (trailing && (*trailing !='\0' && *trailing !='\n' )) { + ret =fsck_refs_report (o,NULL , refname.buf,NULL , +FSCK_MSG_TRAILING_REF_CONTENT , +"trailing garbage in ref" ); + goto clean; + } + } + +clean: +strbuf_release (&abs_gitdir); +strbuf_release (&pointee_path); +strbuf_release (&refname); +strbuf_release (&ref_content); +strbuf_release (&referent); +return ret; +} + static int files_fsck_refs_dir(struct ref_store *ref_store, struct fsck_options *o, const char *refs_check_dir,@@ -3490,6 +3630,7 @@ static int files_fsck_refs(struct ref_store *ref_store,int ret;files_fsck_refs_fn fsck_refs_fns[]= { files_fsck_refs_name, + files_fsck_refs_content,NULL };refs/refs-internal.h +3
-2@@ -709,11 +709,12 @@ struct ref_store {/* * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for -* invalid contents. +* invalid contents. Also *trailing is set to the first character after the +* refname or NULL if the referent is not empty. */ int parse_loose_ref_contents (const char *buf,struct object_id *oid,struct strbuf *referent,unsigned int *type, -int *failure_errno); +int *failure_errno,const char **trailing);/* * Fill in the generic part of refs and add it to our collection of t/t0602-reffiles-fsck.sh +110
-1@@ -98,4 +98,114 @@ test_expect_success 'ref name check should be adapted into fsck messages' ' ) ' +test_expect_success 'regular ref content should be checked' ' + test_when_finished "rm -rf repo" && + git init repo && + branch_dir_prefix=.git/refs/heads && + tag_dir_prefix=.git/refs/tags && + ( + cd repo && + git commit --allow-empty -m initial && + git checkout -b branch-1 && + git tag tag-1 && + git commit --allow-empty -m second && + git checkout -b branch-2 && + git tag tag-2 && + git checkout -b a/b/tag-2 + ) && + ( + cd repo && + printf "%s garbage" "$(git rev-parse branch-1)" > $branch_dir_prefix/branch-1-garbage && + git fsck 2>err && + cat >expect <<-EOF && + warning: refs/heads/branch-1-garbage: trailingRefContent: trailing garbage in ref + EOF + rm $branch_dir_prefix/branch-1-garbage && + test_cmp expect err + ) && + ( + cd repo && + printf "%s garbage" "$(git rev-parse tag-1)" > $tag_dir_prefix/tag-1-garbage && + test_must_fail git -c fsck.trailingRefContent=error fsck 2>err && + cat >expect <<-EOF && + error: refs/tags/tag-1-garbage: trailingRefContent: trailing garbage in ref + EOF + rm $tag_dir_prefix/tag-1-garbage && + test_cmp expect err + ) && + ( + cd repo && + printf "%s " "$(git rev-parse tag-2)" > $tag_dir_prefix/tag-2-garbage && + git fsck 2>err && + cat >expect <<-EOF && + warning: refs/tags/tag-2-garbage: trailingRefContent: trailing garbage in ref + EOF + rm $tag_dir_prefix/tag-2-garbage && + test_cmp expect err + ) && + ( + cd repo && + printf "xfsazqfxcadas" > $tag_dir_prefix/tag-2-bad && + test_must_fail git refs verify 2>err && + cat >expect <<-EOF && + error: refs/tags/tag-2-bad: badRefContent: invalid ref content + EOF + rm $tag_dir_prefix/tag-2-bad && + test_cmp expect err + ) && + ( + cd repo && + printf "xfsazqfxcadas" > $branch_dir_prefix/a/b/branch-2-bad && + test_must_fail git refs verify 2>err && + cat >expect <<-EOF && + error: refs/heads/a/b/branch-2-bad: badRefContent: invalid ref content + EOF + rm $branch_dir_prefix/a/b/branch-2-bad && + test_cmp expect err + ) +' + +test_expect_success 'symbolic ref content should be checked' ' + test_when_finished "rm -rf repo" && + git init repo && + branch_dir_prefix=.git/refs/heads && + tag_dir_prefix=.git/refs/tags && + ( + cd repo && + git commit --allow-empty -m initial && + git checkout -b branch-1 && + git tag tag-1 + ) && + ( + cd repo && + printf "ref: refs/heads/.branch" > $branch_dir_prefix/branch-2-bad && + test_must_fail git refs verify 2>err && + cat >expect <<-EOF && + error: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname + EOF + rm $branch_dir_prefix/branch-2-bad && + test_cmp expect err + ) && + ( + cd repo && + printf "ref: refs/heads" > $branch_dir_prefix/branch-2-bad && + test_must_fail git refs verify 2>err && + cat >expect <<-EOF && + error: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target + EOF + rm $branch_dir_prefix/branch-2-bad && + test_cmp expect err + ) && + ( + cd repo && + printf "ref: logs/maint-v2.45" > $branch_dir_prefix/branch-2-bad && + test_must_fail git refs verify 2>err && + cat >expect <<-EOF && + error: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy + EOF + rm $branch_dir_prefix/branch-2-bad && + test_cmp expect err + ) +' + test_done --2.45.2
Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend
Quoted 7 lines · Karthik Nayak
shejialuo <[email protected]> writes:
... 260+ lines of the original removed ...Review
Quoted 1 line
(quoted diff) +1
-0+ */What happens if a symbolic reference has trailing garbage ?
... 160+ lines of the original removed ...
It is a pain to have to look for only a single line with a new piece of information in a 400+ line response, more than 99% of which are quoted original. Can you trim your quote a bit better?
Thanks.
Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend
Quoted 8 lines · Junio C Hamano
Quoted 6 lines · Karthik Nayak
shejialuo <[email protected]> writes:
... 260+ lines of the original removed ...Review
Quoted 1 line
(quoted diff) +1
-0+ */What happens if a symbolic reference has trailing garbage ?
Quoted 7 lines
Quoted 1 line
... 160+ lines of the original removed ...
It is a pain to have to look for only a single line with a new piece of information in a 400+ line response, more than 99% of which are quoted original. Can you trim your quote a bit better?
Thanks.
I usually do, I missed out here! Sorry about that.
Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend
In reply to Karthik NayakQuoted 19 lines · Karthik Nayak
Quoted 6 lines · shejialuo
refs/files-backend.c +6
-0+/* +* If the ref is a symref, we need to check the destination name and +* connectivity. +*/ +if (referent.len && (type &REF_ISSYMREF )) { +strbuf_addf (&pointee_path,"%s/%s" , gitdir, referent.buf);refs/files-backend.c static int files_fsck_refs_name(struct fsck_options *o, · line 3563
Quoted 10 lines
refs/files-backend.c +10
-0+strbuf_rtrim (&referent); + + ret =files_fsck_symref_target (o, refname.buf, referent.buf, + pointee_path.buf); + goto clean; + }else { +/* +* Only regular refs could have a trailing garbage. Should +* be reported as a warning. +*/
Quoted 2 lines
What happens if a symbolic reference has trailing garbage ?
The "parse_loose_ref_contents" will return the referent. In this function, it will skip the prefix "ref:" to get the pointee. If there are some trailing garbage, it will be reported by the "files_fsck_symref_target".
"files_fsck_symref_target" will use "check_refname_format" function to check the pointee. For example, if the content is "ref: refs/heads/ master garbage". The "refs/heads/master garbage" is a bad name.
However, in my design, the trailing spaces or newline will be ignored, I thought we may not report this problem. And I use "strbuf_rtrim" here to ignore spaces and newlines.
And I think there are some differences between symbolic refs and regular refs when parsing. For regular refs, git will ignore any trailing garbage, however for symbolic refs, git will only ignore the newlines and spaces garbage. And git will not parse "refs/heads/master gar", it's an error here. But for regular refs, for example "edaca... garbage", git will parse it normally without any warnings.
So question comes here, should we warn the user about the trailing newlines or spaces. When using "git symbolic-ref refs/heads/maint refs/heads/main", the "refs/heads/maint" will contain the newline '\n' here and git also accepts content without newline '\n'. And I think we should not warn the user about one newline or no newline.
In my opinion, we should do this. It's not hard to do that. We only warn the user for the following two situations:
1. two or more newlines.
2. one or more spaces.
I will improve code in the next version.
Thanks,
Jialuo
Re: [GSoC][PATCH v11 10/10] fsck: add ref content check for files backend
Quoted 26 lines
The "
parse_loose_ref_contents" will return the referent. In this function, it will skip the prefix "ref:" to get the pointee. If there are some trailing garbage, it will be reported by the "files_fsck_symref_target"."
files_fsck_symref_target" will use "check_refname_format" function to check the pointee. For example, if the content is "ref: refs/heads/ master garbage". The "refs/heads/master garbage" is a bad name.However, in my design, the trailing spaces or newline will be ignored, I thought we may not report this problem. And I use "
strbuf_rtrim" here to ignore spaces and newlines.And I think there are some differences between symbolic refs and regular refs when parsing. For regular refs, git will ignore any trailing garbage, however for symbolic refs, git will only ignore the newlines and spaces garbage. And git will not parse "refs/heads/master gar", it's an error here. But for regular refs, for example "edaca... garbage", git will parse it normally without any warnings.
So question comes here, should we warn the user about the trailing newlines or spaces. When using "git symbolic-ref refs/heads/maint refs/heads/main", the "refs/heads/maint" will contain the newline '\n' here and git also accepts content without newline '\n'. And I think we should not warn the user about one newline or no newline.
Quoted 8 lines
In my opinion, we should do this. It's not hard to do that. We only warn the user for the following two situations:
1. two or more newlines.
2. one or more spaces.I will improve code in the next version.
Well, I have changed my idea here. Sorry for that. If we report this as warning, what is the difference between the following two cases:
1. "ref: refs/heads/master garbage "
2. "ref: refs/heads/master garbage"
For 1, if we do this, we will first report it has the extra spaces and then report it's a bad refname. For 2, we will report it's a bad refname. I think we should not report extra spaces or newlines here even with the error type "warning".
v12 cover ref consistency check infra setup
In reply to shejialuoHi All:
This version handles the following problem:
In order to provide more extensibility, I follow the Karthik's advice using the following data structure:
struct fsck_refs_info {
char *ref_checkee;
union {
struct {
char *sub_ref_checkee;
} files;
};
};Because we use a struct here, we could add more fields when we want to implement morec checks without changing the existing functions. And I move this structure into the "fsck_options". Thus there is no need for us to change the "error_func" prototype which makes this series more clearer.
And Karthik asked me why I does not handle the trailing garbage for symbolic refs. And I cited my understanding here:
Quoted 11 lines
The "
parse_loose_ref_contents" will return the referent. In this function, it will skip the prefix "ref:" to get the pointee. If there are some trailing garbage, it will be reported by the "files_fsck_symref_target"."
files_fsck_symref_target" will use "check_refname_format" function to check the pointee. For example, if the content is "ref: refs/heads/ master garbage". The "refs/heads/master garbage" is a bad name.However, in my design, the trailing spaces or newline will be ignored, I thought we may not report this problem. And I use "
strbuf_rtrim" here to ignore spaces and newlines.
We should not report this even with warning in my perspecitve.
CI: https://github.com/shejialuo/git/pull/10
shejialuo (10):
fsck: rename "skiplist" to "skip_oids"
fsck: add a unified interface for reporting fsck messages
fsck: rename objects-related fsck error functions
fsck: add refs-related error report function
refs: set up ref consistency check infrastructure git refs: add verify subcommand
builtin/fsck: add git-refs verify child process
files-backend: add unified interface for refs scanning
fsck: add ref name check for files backend
fsck: add ref content check for files backend
Documentation/fsck-msgids.txt | 12 ++ Documentation/git-refs.txt | 13 ++ builtin/fsck.c | 31 ++++- builtin/refs.c | 44 ++++++ fsck.c | 106 +++++++++++--- fsck.h | 74 +++++++--- refs.c | 7 +- refs.h | 8 ++ refs/debug.c | 11 ++ refs/files-backend.c | 253 +++++++++++++++++++++++++++++++++- refs/packed-backend.c | 8 ++ refs/refs-internal.h | 11 +- refs/reftable-backend.c | 8 ++ t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++ 14 files changed, 746 insertions(+), 51 deletions(-) create mode 100755 t/t0602-reffiles-fsck.sh
Range-diff against v11:
1: a69705b777 = 1: a69705b777 fsck: rename "skiplist" to "skip_oids" 2: 1ef1036348 < -: ---------- fsck: add a unified interface for reporting fsck messages -: ---------- > 2: a4bfccd938 fsck: add a unified interface for reporting fsck messages 3: d17cf6166e ! 3: 9bc8892761 fsck: rename objects-related fsck error functions
@@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)
-static int fsck_error_func(struct fsck_options *o UNUSED,
- const struct object_id *oid,
- enum object_type object_type,
-- const char *ref_checkee UNUSED,
-- const char *sub_ref_checkee UNUSED,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+static int fsck_objects_error_func(struct fsck_options *o UNUSED,
+ const struct object_id *oid,
+ enum object_type object_type,
-+ const char *ref_checkee UNUSED,
-+ const char *sub_ref_checkee UNUSED,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
@@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,
-int fsck_error_function(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type UNUSED,
-- const char *ref_checkee UNUSED,
-- const char *sub_ref_checkee UNUSED,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+int fsck_objects_error_function(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type UNUSED,
-+ const char *ref_checkee UNUSED,
-+ const char *sub_ref_checkee UNUSED,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
@@ fsck.c: int git_fsck_config(const char *var, const char *value,
-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type,
-- const char *ref_checkee,
-- const char *sub_ref_checkee,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message)
+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
-+ const char *ref_checkee,
-+ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message)
@@ fsck.c: int git_fsck_config(const char *var, const char *value,
puts(oid_to_hex(oid));
return 0;
}
-- return fsck_error_function(o, oid, object_type, ref_checkee,
-- sub_ref_checkee, msg_type, msg_id, message);
-+ return fsck_objects_error_function(o, oid, object_type, ref_checkee,
-+ sub_ref_checkee, msg_type, msg_id,
-+ message);
+- return fsck_error_function(o, oid, object_type, msg_type, msg_id, message);
++ return fsck_objects_error_function(o, oid, object_type,
++ msg_type, msg_id, message);
}
## fsck.h ##
@@ fsck.h: typedef int (*fsck_error)(struct fsck_options *o,
-int fsck_error_function(struct fsck_options *o,
- const struct object_id *oid, enum object_type object_type,
-- const char *ref_checkee, const char *sub_ref_checkee,
- enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
- const char *message);
-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type,
-- const char *ref_checkee,
-- const char *sub_ref_checkee,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message);
+int fsck_objects_error_function(struct fsck_options *o,
+ const struct object_id *oid, enum object_type object_type,
-+ const char *ref_checkee, const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
+ const char *message);
+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
-+ const char *ref_checkee,
-+ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message);
- struct fsck_options {
- fsck_walk_func walk;
+ /*
+ * The information for reporting refs-related error message @@ fsck.h: struct fsck_options {
.gitmodules_done = OIDSET_INIT, \
.gitattributes_found = OIDSET_INIT, \
4: 19e049ee15 ! 4: 82296dc2b9 fsck: add refs-related error report function
@@ Commit message
fsck: add refs-related error report function
Create refs-specific "error_func" callback "fsck_refs_error_function"
- which could provide the following report messages.
+ which could provide the following report messages for files backend
1. "ref_checkee": "fsck error name": "user message".
2. "ref_checkee.sub_ref_checkee": "fsck error name": "user message".
@@ Commit message
message".
"fsck_refs_error_function" uses the "ref_checkee" and "sub_ref_checkee"
- parameters to indicate the information of the checked refs. For loose
- ref and reflog, it only uses the "ref_checkee" parameter. For packed
+ in the "fsck_refs_info" to indicate the information of the checked refs.
+ For loose ref and reflog, it only uses the "ref_checkee". For packed
refs and reftable refs, when checking the consistency of the file
- itself, it still only uses "ref_checkee" parameter. However, when
- checking the consistency of the ref or reflog contained in the file, it
- will use "sub_ref_checkee" parameter to indicate that we are not
- checking the file but the incorporated ref or reflog.
+ itself, it still only uses "ref_checkee". However, when checking the
+ consistency of the ref or reflog contained in the file ,it will use the
+ "sub_ref_checkee" to indicate that we are not checking the file but the
+ incorporated ref or reflog.
"fsck_refs_error_function" will use the "oid" parameter if the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex
@@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,
return 1;
}
-+int fsck_refs_error_function(struct fsck_options *options UNUSED, ++int fsck_refs_error_function(struct fsck_options *options,
+ const struct object_id *oid,
+ enum object_type object_type UNUSED,
-+ const char *ref_checkee,
-+ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
+{
+ struct strbuf sb = STRBUF_INIT;
++ struct fsck_refs_info *refs_info = &options->refs_info;
+ int ret = 0;
+
-+ if (sub_ref_checkee)
-+ strbuf_addf(&sb, "%s.%s", ref_checkee, sub_ref_checkee);
-+ else
-+ strbuf_addstr(&sb, ref_checkee);
++ if (the_repository->ref_storage_format == REF_STORAGE_FORMAT_FILES) {
++ strbuf_addstr(&sb, refs_info->ref_checkee);
++ if (refs_info->u.files.sub_ref_checkee)
++ strbuf_addf(&sb, ".%s", refs_info->u.files.sub_ref_checkee);
+
-+ if (oid)
-+ strbuf_addf(&sb, " -> (%s)", oid_to_hex(oid));
++ if (oid)
++ strbuf_addf(&sb, " -> (%s)", oid_to_hex(oid));
++ }
+
+ if (msg_type == FSCK_WARN)
+ warning("%s: %s", sb.buf, message);
@@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *
+int fsck_refs_error_function(struct fsck_options *options,
+ const struct object_id *oid,
+ enum object_type object_type,
-+ const char *ref_checkee,
-+ const char *sub_ref_checkee,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message);
- struct fsck_options {
- fsck_walk_func walk;
+ /*
+ * The information for reporting refs-related error message @@ fsck.h: struct fsck_options {
.gitattributes_done = OIDSET_INIT, \
.error_func = fsck_objects_error_cb_print_missing_gitmodules, \
5: f175afc37c = 5: c5cac2e318 refs: set up ref consistency check infrastructure
6: e177157faa ! 6: 84d840506e git refs: add verify subcommand
@@ fsck.h: struct fsck_options {
unsigned strict:1;
+ unsigned verbose:1;
enum fsck_msg_type *msg_type;
+ struct fsck_refs_info refs_info;
struct oidset skip_oids;
- struct oidset gitmodules_found;
7: ee0e322f2b = 7: 3fc77ec329 builtin/fsck: add `git-refs verify` child process
8: 6a04fb0170 = 8: 44a75141fa files-backend: add unified interface for refs scanning
9: 7d11836deb ! 9: 4a0d58b07d fsck: add ref name check for files backend
@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,
+
+ if (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {
+ strbuf_addf(&sb, "%s/%s", refs_check_dir, iter->relative_path);
-+ ret = fsck_refs_report(o, NULL, sb.buf, NULL,
++ o->refs_info.ref_checkee = sb.buf;
++ ret = fsck_refs_report(o, NULL,
+ FSCK_MSG_BAD_REF_NAME,
+ "invalid refname format");
+ }
10: ad696852ba ! 10: c529670e54 fsck: add ref content check for files backend
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+
+ if (!skip_prefix(pointee_name, "refs/", &p)) {
+
-+ ret = fsck_refs_report(o, NULL, refname, NULL,
++ ret = fsck_refs_report(o, NULL,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
+ "point to target out of refs hierarchy");
+ goto out;
+ }
+
+ if (check_refname_format(pointee_name, 0)) {
-+ ret = fsck_refs_report(o, NULL, refname, NULL,
++ ret = fsck_refs_report(o, NULL,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
+ "point to invalid refname");
+ }
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ goto out;
+
+ if (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {
-+ ret = fsck_refs_report(o, NULL, refname, NULL,
++ ret = fsck_refs_report(o, NULL,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
+ "point to invalid target");
+ goto out;
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ int ret = 0;
+
+ strbuf_addf(&refname, "%s/%s", refs_check_dir, iter->relative_path);
++ o->refs_info.ref_checkee = refname.buf;
+
+ /*
+ * If the file is a symlink, we need to only check the connectivity
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+
+ if (!skip_prefix(pointee_path.buf,
+ abs_gitdir.buf, &pointee_name)) {
-+ ret = fsck_refs_report(o, NULL, refname.buf, NULL,
++ ret = fsck_refs_report(o, NULL,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
+ "point to target outside gitdir");
+ goto clean;
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ if (parse_loose_ref_contents(ref_content.buf, &oid,
+ &referent, &type,
+ &failure_errno, &trailing)) {
-+ ret = fsck_refs_report(o, NULL, refname.buf, NULL,
++ ret = fsck_refs_report(o, NULL,
+ FSCK_MSG_BAD_REF_CONTENT,
+ "invalid ref content");
+ goto clean;
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ pointee_path.buf);
+ goto clean;
+ } else {
-+ /*
-+ * Only regular refs could have a trailing garbage. Should
-+ * be reported as a warning.
-+ */
+ if (trailing && (*trailing != '\0' && *trailing != '\n')) {
-+ ret = fsck_refs_report(o, NULL, refname.buf, NULL,
++ ret = fsck_refs_report(o, NULL,
+ FSCK_MSG_TRAILING_REF_CONTENT,
+ "trailing garbage in ref");
+ goto clean;Signature · 1 line
2.45.2
PATCH v12 1/10 fsck: rename "skiplist" to "skip_oids"
The "skiplist" field in "fsck_options" is related to objects. Because we are going to introduce ref consistency check, the "skiplist" name is too general which will make the caller think "skiplist" is related to both the refs and objects.
It may seem that for both refs and objects, we should provide a general "skiplist" here. However, the type for "skiplist" is struct oidset which is totally unsuitable for refs.
To avoid above ambiguity, rename "skiplist" to "skip_oids".
--- fsck.c | 4 ++-- fsck.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
fsck.c +2 -2
@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)if (!strcmp (buf,"skiplist" )) {if (equal == len)die ("skiplist requires a path" ); -oidset_parse_file (&options->skiplist, buf + equal +1 , +oidset_parse_file (&options->skip_oids, buf + equal +1 , the_repository->hash_algo); buf += len +1 ;continue ;
@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)static int object_on_skiplist (struct fsck_options *opts,const struct object_id *oid) { -return opts && oid &&oidset_contains (&opts->skiplist, oid); +return opts && oid &&oidset_contains (&opts->skip_oids, oid); } __attribute__((format (printf, 5, 6)))
fsck.h +2 -2
@@ -136,7 +136,7 @@ struct fsck_options {fsck_error error_func; unsigned strict:1;enum fsck_msg_type *msg_type; -struct oidset skiplist; +struct oidset skip_oids;struct oidset gitmodules_found;struct oidset gitmodules_done;struct oidset gitattributes_found;
@@ -145,7 +145,7 @@ struct fsck_options { };#define FSCK_OPTIONS_DEFAULT { \ - .skiplist = OIDSET_INIT, \ + .skip_oids = OIDSET_INIT, \ .gitmodules_found = OIDSET_INIT, \ .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \
Signature · 1 line
2.45.2
PATCH v12 2/10 fsck: add a unified interface for reporting fsck messages
In reply to shejialuoThe static function "report" provided by "fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "fsck_vreport" following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.
In order to provide an extensible error report for refs, add a new "fsck_refs_info" structure to incorporate an union for supporting different checks for files backend and reftable backend. Then incorporate this structure into the "fsck_options" to reuse the "error_func" callback.
Then, change "report" function to use "fsck_vreport" to report objects related messages. Add a new function called "fsck_refs_report" to use "fsck_vreport" to report refs related messages.
--- fsck.c | 46 +++++++++++++++++++++++++++++++++++++++------- fsck.h | 26 +++++++++++++++++++++++++- 2 files changed, 64 insertions(+), 8 deletions(-)
fsck.c +39 -7
@@ -226,12 +226,17 @@ static int object_on_skiplist(struct fsck_options *opts,return opts && oid &&oidset_contains (&opts->skip_oids, oid); } -__attribute__((format (printf,5 ,6 ))) -static int report (struct fsck_options *options, -const struct object_id *oid,enum object_type object_type, -enum fsck_msg_id msg_id,const char *fmt, ...) +/* +* Provide a unified interface for either fscking refs or objects. +* It will get the current msg error type and call the error_func callback +* which is registered in the "fsck_options" struct. +*/ +static int fsck_vreport (struct fsck_options *options, +const struct object_id *oid, +enum object_type object_type, +enum fsck_msg_id msg_id,const char *fmt,va_list ap) { -va_list ap; +va_list ap_copy;struct strbuf sb =STRBUF_INIT ;enum fsck_msg_type msg_type =fsck_msg_type (msg_id, options);int result;
@@ -250,8 +255,8 @@ static int report(struct fsck_options *options,prepare_msg_ids ();strbuf_addf (&sb,"%s: " , msg_id_info[msg_id].camelcased); -va_start (ap, fmt); -strbuf_vaddf (&sb, fmt, ap); +va_copy (ap_copy, ap); +strbuf_vaddf (&sb, fmt, ap_copy); result = options->error_func (options, oid, object_type, msg_type, msg_id, sb.buf);strbuf_release (&sb);
@@ -260,6 +265,33 @@ static int report(struct fsck_options *options,return result; } +__attribute__((format (printf,5 ,6 ))) +static int report (struct fsck_options *options, +const struct object_id *oid,enum object_type object_type, +enum fsck_msg_id msg_id,const char *fmt, ...) +{ +va_list ap; +int result; + +va_start (ap, fmt); + result =fsck_vreport (options, oid, object_type, msg_id, fmt, ap); +va_end (ap); + +return result; +} + +int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +enum fsck_msg_id msg_id,const char *fmt, ...) +{ +va_list ap; +int result; +va_start (ap, fmt); + result =fsck_vreport (options, oid,OBJ_NONE , msg_id, fmt, ap); +va_end (ap); +return result; +} +void fsck_enable_object_names (struct fsck_options *options) {if (!options->object_names)
fsck.h +25 -1
@@ -114,7 +114,9 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);typedef int (*fsck_walk_func )(struct object *obj,enum object_type object_type,void *data,struct fsck_options *options); -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */ +/* +* callback function for reporting errors when checking either objects or refs +*/ typedef int (*fsck_error)(struct fsck_options *o, const struct object_id *oid, enum object_type object_type, enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
@@ -131,11 +133,24 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,enum fsck_msg_id msg_id, const char *message); +/* +* The information for reporting refs-related error message +*/ +struct fsck_refs_info { +char *ref_checkee; +union { +struct { +char *sub_ref_checkee; + } files; + } u; +}; +struct fsck_options {fsck_walk_func walk;fsck_error error_func;unsigned strict:1 ;enum fsck_msg_type *msg_type; +struct fsck_refs_info refs_info;struct oidset skip_oids;struct oidset gitmodules_found;struct oidset gitmodules_done;
@@ -209,6 +224,15 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +/* +* Report an error or warning for refs. +*/ +__attribute__((format (printf,4 ,5 ))) +int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +enum fsck_msg_id msg_id, +const char *fmt, ...); +/* * Subsystem for storing human-readable names for each object. *
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v12 02/10] fsck: add a unified interface for reporting fsck messages
shejialuo <[email protected]> writes:
[snip]
Quoted 21 lines
fsck.h +5
-1@@ -114,7 +114,9 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);typedef int (*fsck_walk_func )(struct object *obj,enum object_type object_type,void *data,struct fsck_options *options); -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */ +/* +* callback function for reporting errors when checking either objects or refs +*/ typedef int (*fsck_error)(struct fsck_options *o, const struct object_id *oid, enum object_type object_type, enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,@@ -131,11 +133,24 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,enum fsck_msg_id msg_id, const char *message); +/* +* The information for reporting refs-related error message
fsck.h int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o, · line 147
Quoted 10 lines
fsck.h +10
-0+*/ +struct fsck_refs_info { +char *ref_checkee; +union { +struct { +char *sub_ref_checkee; + } files; + } u; +}; +
When I suggested using a struct, it was to replace sending both 'ref_checkee' and 'sub_ref_checkee' to the error functions. I.e to send a single struct instead of two different fields. But now, we've included it in the 'fsck_options' struct, I don't think it should be part of the options. Because the fsck_options is used beyond the error function while 'fsck_refs_info' is only needed in the error functions, right?
PATCH v12 3/10 fsck: rename objects-related fsck error functions
In reply to shejialuoThe names of objects-related fsck error functions are general. It's OK when there is only object database check. However, we have introduced refs database check report function. To avoid ambiguity, rename object-related fsck error functions to explicitly indicate these functions are used to report objects-related messages.
--- builtin/fsck.c | 14 +++++++------- fsck.c | 27 ++++++++++++++------------- fsck.h | 26 +++++++++++++------------- 3 files changed, 34 insertions(+), 33 deletions(-)
builtin/fsck.c +7 -7
@@ -89,12 +89,12 @@ static int objerror(struct object *obj, const char *err)return -1 ; } -static int fsck_error_func(struct fsck_options *o UNUSED, -const struct object_id *oid, -enum object_type object_type, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +static int fsck_objects_error_func(struct fsck_options *o UNUSED, +const struct object_id *oid, +enum object_type object_type, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {switch (msg_type) {case FSCK_WARN :
@@ -938,7 +938,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) fsck_walk_options.walk = mark_object; fsck_obj_options.walk = mark_used; - fsck_obj_options.error_func = fsck_error_func; + fsck_obj_options.error_func = fsck_objects_error_func;if (check_strict) fsck_obj_options.strict =1 ;
fsck.c +14 -13
@@ -1232,12 +1232,12 @@ int fsck_buffer(const struct object_id *oid, enum object_type type, type); } -int fsck_error_function (struct fsck_options *o, -const struct object_id *oid, - enum object_type object_type UNUSED, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +int fsck_objects_error_function (struct fsck_options *o, +const struct object_id *oid, + enum object_type object_type UNUSED, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {if (msg_type ==FSCK_WARN ) {warning ("object %s: %s" ,fsck_describe_object (o, oid), message);
@@ -1335,16 +1335,17 @@ int git_fsck_config(const char *var, const char *value,* Custom error callbacks that are used in more than one place. */ -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message) +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +const struct object_id *oid, +enum object_type object_type, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message) {if (msg_id ==FSCK_MSG_GITMODULES_MISSING ) {puts (oid_to_hex (oid));return 0 ; } -return fsck_error_function (o, oid, object_type, msg_type, msg_id, message); +return fsck_objects_error_function (o, oid, object_type, + msg_type, msg_id, message); }
fsck.h +13 -13
@@ -122,16 +122,16 @@ typedef int (*fsck_error)(struct fsck_options *o, enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); -int fsck_error_function (struct fsck_options *o, -const struct object_id *oid,enum object_type object_type, -enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, -const char *message); -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message); +int fsck_objects_error_function (struct fsck_options *o, +const struct object_id *oid,enum object_type object_type, +enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, +const char *message); +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +const struct object_id *oid, +enum object_type object_type, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);/* * The information for reporting refs-related error message
@@ -165,7 +165,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function \ + .error_func = fsck_objects_error_function \ }#define FSCK_OPTIONS_STRICT { \ .strict = 1, \
@@ -173,7 +173,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function, \ + .error_func = fsck_objects_error_function, \ }#define FSCK_OPTIONS_MISSING_GITMODULES { \ .strict = 1, \
@@ -181,7 +181,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_cb_print_missing_gitmodules, \ + .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ }/* descend in all linked child objects
Signature · 1 line
2.45.2
PATCH v12 4/10 fsck: add refs-related error report function
In reply to shejialuoCreate refs-specific "error_func" callback "fsck_refs_error_function" which could provide the following report messages for files backend
1. "ref_checkee": "fsck error name": "user message".
2. "ref_checkee.sub_ref_checkee": "fsck error name": "user message".
3. "ref_checkee -> (oid hex)": "fsck error name": "user message".
4. "ref_checkee.sub_ref_checkee -> (oid hex)": "fsck error name": "user message".
"fsck_refs_error_function" uses the "ref_checkee" and "sub_ref_checkee" in the "fsck_refs_info" to indicate the information of the checked refs. For loose ref and reflog, it only uses the "ref_checkee". For packed refs and reftable refs, when checking the consistency of the file itself, it still only uses "ref_checkee". However, when checking the consistency of the ref or reflog contained in the file ,it will use the "sub_ref_checkee" to indicate that we are not checking the file but the incorporated ref or reflog.
"fsck_refs_error_function" will use the "oid" parameter if the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.
Last, add "FSCK_REFS_OPTIONS_DEFAULT" and "FSCK_REFS_OPTIONS_STRICT" macros to create refs options easily.
--- fsck.c | 29 +++++++++++++++++++++++++++++ fsck.h | 13 +++++++++++++ 2 files changed, 42 insertions(+)
fsck.c +29 -0
@@ -1247,6 +1247,35 @@ int fsck_objects_error_function(struct fsck_options *o,return 1 ; } +int fsck_refs_error_function (struct fsck_options *options, +const struct object_id *oid, + enum object_type object_type UNUSED, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) +{ +struct strbuf sb =STRBUF_INIT ; +struct fsck_refs_info *refs_info = &options->refs_info; +int ret =0 ; + +if (the_repository->ref_storage_format ==REF_STORAGE_FORMAT_FILES ) { +strbuf_addstr (&sb, refs_info->ref_checkee); +if (refs_info->u.files.sub_ref_checkee) +strbuf_addf (&sb,".%s" , refs_info->u.files.sub_ref_checkee); + +if (oid) +strbuf_addf (&sb," -> (%s)" ,oid_to_hex (oid)); + } + +if (msg_type ==FSCK_WARN ) +warning ("%s: %s" , sb.buf, message); +else + ret =error ("%s: %s" , sb.buf, message); + +strbuf_release (&sb); +return ret; +} +static int fsck_blobs (struct oidset *blobs_found,struct oidset *blobs_done,enum fsck_msg_id msg_missing,enum fsck_msg_id msg_type,struct fsck_options *options,const char *blob_type)
fsck.h +13 -0
@@ -132,6 +132,12 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); +int fsck_refs_error_function (struct fsck_options *options, +const struct object_id *oid, +enum object_type object_type, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);/* * The information for reporting refs-related error message
@@ -183,6 +189,13 @@ struct fsck_options { .gitattributes_done = OIDSET_INIT, \ .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ } +#define FSCK_REFS_OPTIONS_DEFAULT { \ + .error_func = fsck_refs_error_function, \ +} +#define FSCK_REFS_OPTIONS_STRICT { \ + .strict = 1, \ + .error_func = fsck_refs_error_function, \ +}/* descend in all linked child objects * the return value is:
Signature · 1 line
2.45.2
PATCH v12 5/10 refs: set up ref consistency check infrastructure
In reply to shejialuoThe interfaces defined in the ref_storage_be are carefully structured in semantic. It's organized as the five parts:
1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.
To keep consistent with the git-fsck(1), add a new interface named "fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.
Last, implement placeholder functions for each ref backends.
--- refs.c | 5 +++++ refs.h | 8 ++++++++ refs/debug.c | 11 +++++++++++ refs/files-backend.c | 15 ++++++++++++++- refs/packed-backend.c | 8 ++++++++ refs/refs-internal.h | 6 ++++++ refs/reftable-backend.c | 8 ++++++++ 7 files changed, 60 insertions(+), 1 deletion(-)
refs.c +5 -0
@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)return check_or_sanitize_refname (refname, flags,NULL ); } +int refs_fsck (struct ref_store *refs,struct fsck_options *o) +{ +return refs->be->fsck (refs, o); +} +void sanitize_refname_component (const char *refname,struct strbuf *out) {if (check_or_sanitize_refname (refname,REFNAME_ALLOW_ONELEVEL , out))
refs.h +8 -0
@@ -4,6 +4,7 @@#include "commit.h" #include "repository.h" +struct fsck_options ;struct object_id ;struct ref_store ;struct strbuf ;
@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat*/ int check_refname_format (const char *refname,int flags); +/* +* Check the reference database for consistency. Return 0 if refs and +* reflogs are consistent, and non-zero otherwise. The errors will be +* written to stderr. +*/ +int refs_fsck (struct ref_store *refs,struct fsck_options *o); +/* * Apply the rules from check_refname_format, but mutate the result until it * is acceptable, and place the result in "out".
refs/debug.c +11 -0
@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,return res; } +static int debug_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store; +int res = drefs->refs->be->fsck (drefs->refs, o); +trace_printf_key (&trace_refs,"fsck: %d\n" , res); +return res; +} +struct ref_storage_be refs_be_debug = { .name ="debug" , .init =NULL ,
@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = { .create_reflog = debug_create_reflog, .delete_reflog = debug_delete_reflog, .reflog_expire = debug_reflog_expire, + + .fsck = debug_fsck, };
refs/files-backend.c +14 -1
@@ -3408,6 +3408,17 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +static int files_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +int ret; +struct files_ref_store *refs = +files_downcast (ref_store,REF_STORE_READ ,"fsck" ); + + ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); +return ret; +} +struct ref_storage_be refs_be_files = { .name ="files" , .init = files_ref_store_init,
@@ -3434,5 +3445,7 @@ struct ref_storage_be refs_be_files = { .reflog_exists = files_reflog_exists, .create_reflog = files_create_reflog, .delete_reflog = files_delete_reflog, - .reflog_expire = files_reflog_expire + .reflog_expire = files_reflog_expire, + + .fsck = files_fsck, };
refs/packed-backend.c +8 -0
@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_sreturn empty_ref_iterator_begin (); } +static int packed_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_packed = { .name ="packed" , .init = packed_ref_store_init,
@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = { .create_reflog = NULL, .delete_reflog = NULL, .reflog_expire = NULL, + + .fsck = packed_fsck, };
refs/refs-internal.h +6 -0
@@ -4,6 +4,7 @@#include "refs.h" #include "iterator.h" +struct fsck_options ;struct ref_transaction ;/*
@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,typedef int read_symbolic_ref_fn (struct ref_store *ref_store,const char *refname,struct strbuf *referent); +typedef int fsck_fn (struct ref_store *ref_store, +struct fsck_options *o); +struct ref_storage_be {const char *name;ref_store_init_fn *init;
@@ -677,6 +681,8 @@ struct ref_storage_be {create_reflog_fn *create_reflog;delete_reflog_fn *delete_reflog;reflog_expire_fn *reflog_expire; + +fsck_fn *fsck; };extern struct ref_storage_be refs_be_files;
refs/reftable-backend.c +8 -0
@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,return ret; } +static int reftable_be_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_reftable = { .name ="reftable" , .init = reftable_be_init,
@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = { .create_reflog = reftable_be_create_reflog, .delete_reflog = reftable_be_delete_reflog, .reflog_expire = reftable_be_reflog_expire, + + .fsck = reftable_be_fsck, };
Signature · 1 line
2.45.2
PATCH v12 6/10 git refs: add verify subcommand
In reply to shejialuoIntroduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency and also this subcommand will be used as the entry point of checking refs for "git-fsck(1)". Last, add "verbose" field into "fsck_options" to indicate whether we should print verbose messages when checking refs and objects consistency.
--- Documentation/git-refs.txt | 13 +++++++++++ builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++ fsck.h | 1 + 3 files changed, 58 insertions(+)
Documentation/git-refs.txt +13 -0
@@ -10,6 +10,7 @@ SYNOPSIS -------- [verse] 'git refs migrate' --ref-format=<format> [--dry-run] +'git refs verify' [--strict] [--verbose] DESCRIPTION -----------
@@ -22,6 +23,9 @@ COMMANDS migrate:: Migrate ref store between different formats. +verify:: + Verify reference database consistency. + OPTIONS -------
@@ -39,6 +43,15 @@ include::ref-storage-format.txt[] can be used to double check that the migration works as expected before performing the actual migration. +The following options are specific to 'git refs verify': + +--strict:: + Enable more strict checking, every WARN severity for the `Fsck Messages` + be seen as ERROR. See linkgit:git-fsck[1]. + +--verbose:: + When verifying the reference database consistency, be chatty. + KNOWN LIMITATIONS -----------------
builtin/refs.c +44 -0
69 lines not on this page · raw message
fsck.h +1 -0
@@ -155,6 +155,7 @@ struct fsck_options {fsck_walk_func walk;fsck_error error_func; unsigned strict:1; + unsigned verbose:1;enum fsck_msg_type *msg_type;struct fsck_refs_info refs_info;struct oidset skip_oids;
Signature · 1 line
2.45.2
PATCH v12 7/10 builtin/fsck: add `git-refs verify` child process
In reply to shejialuoIntroduce a new function "fsck_refs" that initializes and runs a child process to execute the "git-refs verify" command.
--- builtin/fsck.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
builtin/fsck.c +17 -0
@@ -896,6 +896,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)return res; } +static void fsck_refs (void ) +{ +struct child_process refs_verify =CHILD_PROCESS_INIT ; +child_process_init (&refs_verify); + refs_verify.git_cmd =1 ; +strvec_pushl (&refs_verify.args,"refs" ,"verify" ,NULL ); +if (verbose) +strvec_push (&refs_verify.args,"--verbose" ); +if (check_strict) +strvec_push (&refs_verify.args,"--strict" ); + +if (run_command (&refs_verify)) + errors_found |=ERROR_REFS ; +} +static char const *const fsck_usage[] = { N_("git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\n" " [--[no-]full] [--strict] [--verbose] [--lost-found]\n"
@@ -1065,6 +1080,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)check_connectivity (); +fsck_refs (); +if (the_repository->settings.core_commit_graph) {struct child_process commit_graph_verify =CHILD_PROCESS_INIT ;
Signature · 1 line
2.45.2
PATCH v12 8/10 files-backend: add unified interface for refs scanning
In reply to shejialuoFor refs and reflogs, we need to scan its corresponding directories to check every regular file or symbolic link which shares the same pattern. Introduce a unified interface for scanning directories for files-backend.
--- refs/files-backend.c | 77 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 76 insertions(+), 1 deletion(-)
refs/files-backend.c +76 -1
@@ -6,6 +6,7 @@#include "../gettext.h" #include "../hash.h" #include "../hex.h" +#include "../fsck.h" #include "../refs.h" #include "refs-internal.h" #include "ref-cache.h"
@@ -3408,6 +3409,78 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +/* +* For refs and reflogs, they share a unified interface when scanning +* the whole directory. This function is used as the callback for each +* regular file or symlink in the directory. +*/ +typedef int (*files_fsck_refs_fn )(struct fsck_options *o, +const char *gitdir, +const char *refs_check_dir, +struct dir_iterator *iter); + +static int files_fsck_refs_dir (struct ref_store *ref_store, +struct fsck_options *o, +const char *refs_check_dir, +files_fsck_refs_fn *fsck_refs_fns) +{ +const char *gitdir = ref_store->gitdir; +struct strbuf sb =STRBUF_INIT ; +struct dir_iterator *iter; +int iter_status; +int ret =0 ; + +strbuf_addf (&sb,"%s/%s" , gitdir, refs_check_dir); + + iter =dir_iterator_begin (sb.buf,0 ); + +if (!iter) { + ret =error_errno ("cannot open directory %s" , sb.buf); + goto out; + } + +while ((iter_status =dir_iterator_advance (iter)) ==ITER_OK ) { +if (S_ISDIR (iter->st.st_mode)) { +continue ; + }else if (S_ISREG (iter->st.st_mode) || +S_ISLNK (iter->st.st_mode)) { +if (o->verbose) +fprintf_ln (stderr,"Checking %s/%s" , + refs_check_dir, iter->relative_path); +for (size_t i =0 ; fsck_refs_fns[i]; i++) { +if (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter)) + ret =-1 ; + } + }else { + ret =error (_ ("unexpected file type for '%s'" ), + iter->basename); + } + } + +if (iter_status !=ITER_DONE ) + ret =error (_ ("failed to iterate over '%s'" ), sb.buf); + +out: +strbuf_release (&sb); +return ret; +} + +static int files_fsck_refs (struct ref_store *ref_store, +struct fsck_options *o) +{ +int ret; +files_fsck_refs_fn fsck_refs_fns[]= { +NULL + }; + +if (o->verbose) +fprintf_ln (stderr,"Checking references consistency" ); + + ret =files_fsck_refs_dir (ref_store, o,"refs" , fsck_refs_fns); + +return ret; +} +static int files_fsck (struct ref_store *ref_store,struct fsck_options *o) {
@@ -3415,7 +3488,9 @@ static int files_fsck(struct ref_store *ref_store,struct files_ref_store *refs =files_downcast (ref_store,REF_STORE_READ ,"fsck" ); - ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); + ret = refs->packed_ref_store->be->fsck (refs->packed_ref_store, o) + |files_fsck_refs (ref_store, o); +return ret; }
Signature · 1 line
2.45.2
PATCH v12 9/10 fsck: add ref name check for files backend
In reply to shejialuoThe git-fsck(1) only implicitly checks the reference, it does not fully check refs with bad format name such as standalone "@" and name ending with ".lock".
In order to provide such checks, add a new fsck message id "badRefName" with default ERROR type. Use existing "check_refname_format" to explicit check the ref name. And add a new unit test to verify the functionality.
--- Documentation/fsck-msgids.txt | 3 + fsck.h | 1 + refs/files-backend.c | 21 +++++++ t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++ 4 files changed, 126 insertions(+) create mode 100755 t/t0602-reffiles-fsck.sh
Documentation/fsck-msgids.txt +3 -0
@@ -19,6 +19,9 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefName`:: + (ERROR) A ref has a bad name. + `badTagName`:: (INFO) A tag has an invalid format.
fsck.h +1 -0
@@ -31,6 +31,7 @@ enum fsck_msg_type { FUNC(BAD_NAME, ERROR) \ FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ + FUNC(BAD_REF_NAME, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
refs/files-backend.c +21 -0
@@ -3419,6 +3419,26 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,const char *refs_check_dir, struct dir_iterator *iter); +static int files_fsck_refs_name (struct fsck_options *o, + const char *gitdir UNUSED, +const char *refs_check_dir, +struct dir_iterator *iter) +{ +struct strbuf sb =STRBUF_INIT ; +int ret =0 ; + +if (check_refname_format (iter->basename,REFNAME_ALLOW_ONELEVEL )) { +strbuf_addf (&sb,"%s/%s" , refs_check_dir, iter->relative_path); + o->refs_info.ref_checkee = sb.buf; + ret =fsck_refs_report (o,NULL , +FSCK_MSG_BAD_REF_NAME , +"invalid refname format" ); + } + +strbuf_release (&sb); +return ret; +} + static int files_fsck_refs_dir(struct ref_store *ref_store, struct fsck_options *o, const char *refs_check_dir,
@@ -3470,6 +3490,7 @@ static int files_fsck_refs(struct ref_store *ref_store, {int ret;files_fsck_refs_fn fsck_refs_fns[]= { + files_fsck_refs_name,NULL };
t/t0602-reffiles-fsck.sh new file +101 -0
101 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH v12 10/10 fsck: add ref content check for files backend
In reply to shejialuoEnhance the git-fsck(1) command by adding a check for reference content in the files backend. The new functionality ensures that symrefs, real symbolic link and regular refs are validated correctly.
In order to check the trailing content of the regular refs, add a new parameter trailing to parse_loose_ref_contents.
For symrefs, parse_loose_ref_contents will set the "referent". However, symbolic link could be either absolute or relative. Use "strbuf_add_real_path" to read the symbolic link and convert the relative path to absolute path. Then use "skip_prefix" to make it align with symref "referent".
Thus, the symrefs and symbolic links could share the same interface. Add a new function "files_fsck_symref_target" which aims at checking the following things:
1. whether the pointee is under the refs/ directory.
2. whether the pointee name is correct.
3. whether the pointee path is a wrong type in filesystem.
Last, add the following FSCK MESSAGEs:
1. "badRefContent(ERROR)": A ref has a bad content
2. "badSymrefPointee(ERROR)": The pointee of a symref is bad.
3. "trailingRefContent(WARN)": A ref content has trailing contents.
--- Documentation/fsck-msgids.txt | 9 +++ fsck.h | 3 + refs.c | 2 +- refs/files-backend.c | 142 +++++++++++++++++++++++++++++++++- refs/refs-internal.h | 5 +- t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++ 6 files changed, 266 insertions(+), 5 deletions(-)
Documentation/fsck-msgids.txt +9 -0
@@ -19,9 +19,15 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefContent`:: + (ERROR) A ref has a bad content. + `badRefName`:: (ERROR) A ref has a bad name. +`badSymrefPointee`:: + (ERROR) The pointee of a symref is bad. + `badTagName`:: (INFO) A tag has an invalid format.
@@ -167,6 +173,9 @@ `nullSha1`:: (WARN) Tree contains entries pointing to a null sha1. +`trailingRefContent`:: + (WARN) A ref content has trailing contents. + `treeNotSorted`:: (ERROR) A tree is not properly sorted.
fsck.h +3 -0
@@ -32,6 +32,8 @@ enum fsck_msg_type { FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ FUNC(BAD_REF_NAME, ERROR) \ + FUNC(BAD_REF_CONTENT, ERROR) \ + FUNC(BAD_SYMREF_POINTEE, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
@@ -72,6 +74,7 @@ enum fsck_msg_type { FUNC(HAS_DOTDOT, WARN) \ FUNC(HAS_DOTGIT, WARN) \ FUNC(NULL_SHA1, WARN) \ + FUNC(TRAILING_REF_CONTENT, WARN) \ FUNC(ZERO_PADDED_FILEMODE, WARN) \ FUNC(NUL_IN_COMMIT, WARN) \FUNC (LARGE_PATHNAME ,WARN ) \
refs.c +1 -1
@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store, } result =parse_loose_ref_contents (content.buf, oid, referent, type, - failure_errno); + failure_errno,NULL ); done:strbuf_release (&full_path);
refs/files-backend.c +140 -2
178 lines not on this page · raw message
refs/refs-internal.h +3 -2
@@ -709,11 +709,12 @@ struct ref_store {/* * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for -* invalid contents. +* invalid contents. Also *trailing is set to the first character after the +* refname or NULL if the referent is not empty. */ int parse_loose_ref_contents (const char *buf,struct object_id *oid,struct strbuf *referent,unsigned int *type, -int *failure_errno); +int *failure_errno,const char **trailing);/* * Fill in the generic part of refs and add it to our collection of
t/t0602-reffiles-fsck.sh +110 -0
114 lines not on this page · raw message
Signature · 1 line
2.45.2
v13 cover ref consistency check infra setup
In reply to shejialuoHi All:
This version mainly focuses on the problem about how should we provide extensibility for reporting the problem of refs. In patch v10, Junio asked a question here:
Quoted 11 lines
The error reporting function for refs consistency check was still about reporting a problem for a single ref. I am wondering how consistency violations that are not about a single ref should be handled. For example, if refs/packed-backend.c:packed_fsck() finds that the file is not sorted properly or has some unparseable garbage in it, it is not something you can report as "refs/heads/main is broken", but those who are interested in seeing the "reference database consistency" verified, it is very much what they want the tool to notice. How would detection of such a breakage that is not attributed to a single ref fit in this "ref consistency check infrastructure" that was introduced by [05/10]?
Actually, I think that the original parameter "checked_ref_name" is a bad name which makes the reader think that "we only handle refs". And this is my fault for this design. However, I misunderstood the Junio's word and made things complicated.
The patch v11 and v12 wants to solve a problem that we should provide extensibility for reporting refs problem. However, these two versions still made things complicated. After an offline meeting with Patrick and Karthik, we design the following simple flat data structure:
struct fsck_refs_info {
const char *path;
};It is simple and provides extensibility.
Thanks,
Jialuo
shejialuo (10):
fsck: rename "skiplist" to "skip_oids"
fsck: add a unified interface for reporting fsck messages
fsck: rename objects-related fsck error functions
fsck: add refs-related error report function
refs: set up ref consistency check infrastructure git refs: add verify subcommand
builtin/fsck: add git-refs verify child process
files-backend: add unified interface for refs scanning
fsck: add ref name check for files backend
fsck: add ref content check for files backend
Documentation/fsck-msgids.txt | 12 ++ Documentation/git-refs.txt | 13 ++ builtin/fsck.c | 32 ++++- builtin/mktag.c | 1 + builtin/refs.c | 44 ++++++ fsck.c | 109 +++++++++++---- fsck.h | 74 +++++++--- object-file.c | 11 +- refs.c | 7 +- refs.h | 8 ++ refs/debug.c | 11 ++ refs/files-backend.c | 251 +++++++++++++++++++++++++++++++++- refs/packed-backend.c | 8 ++ refs/refs-internal.h | 11 +- refs/reftable-backend.c | 8 ++ t/t0602-reffiles-fsck.sh | 211 ++++++++++++++++++++++++++++ 16 files changed, 754 insertions(+), 57 deletions(-) create mode 100755 t/t0602-reffiles-fsck.sh
Range-diff against v12:
1: a69705b777 = 1: a69705b777 fsck: rename "skiplist" to "skip_oids" 2: a4bfccd938 < -: ---------- fsck: add a unified interface for reporting fsck messages -: ---------- > 2: 178329d085 fsck: add a unified interface for reporting fsck messages 3: 9bc8892761 ! 3: 341cb841bf fsck: rename objects-related fsck error functions
@@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)
-static int fsck_error_func(struct fsck_options *o UNUSED,
- const struct object_id *oid,
- enum object_type object_type,
+- const struct fsck_refs_info *refs_info UNUSED,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+static int fsck_objects_error_func(struct fsck_options *o UNUSED,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const struct fsck_refs_info *refs_info UNUSED,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
@@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,
-int fsck_error_function(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type UNUSED,
+- const struct fsck_refs_info *refs_info UNUSED,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+int fsck_objects_error_function(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type UNUSED,
++ const struct fsck_refs_info *refs_info UNUSED,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
@@ fsck.c: int git_fsck_config(const char *var, const char *value,
-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type,
+- const struct fsck_refs_info *refs_info,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message)
+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const struct fsck_refs_info *refs_info,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message)
@@ fsck.c: int git_fsck_config(const char *var, const char *value,
puts(oid_to_hex(oid));
return 0;
}
-- return fsck_error_function(o, oid, object_type, msg_type, msg_id, message);
-+ return fsck_objects_error_function(o, oid, object_type,
+- return fsck_error_function(o, oid, object_type, refs_info,
+- msg_type, msg_id, message);
++ return fsck_objects_error_function(o, oid, object_type, refs_info,
+ msg_type, msg_id, message);
}
@@ fsck.h: typedef int (*fsck_error)(struct fsck_options *o,
-int fsck_error_function(struct fsck_options *o,
- const struct object_id *oid, enum object_type object_type,
+- const struct fsck_refs_info *refs_info,
- enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
- const char *message);
-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
- const struct object_id *oid,
- enum object_type object_type,
+- const struct fsck_refs_info *refs_info,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message);
+int fsck_objects_error_function(struct fsck_options *o,
+ const struct object_id *oid, enum object_type object_type,
++ const struct fsck_refs_info *refs_info,
+ enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
+ const char *message);
+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const struct fsck_refs_info *refs_info,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message);
4: 82296dc2b9 ! 4: b328f3b218 fsck: add refs-related error report function
@@ Metadata
## Commit message ##
fsck: add refs-related error report function
- Create refs-specific "error_func" callback "fsck_refs_error_function"
- which could provide the following report messages for files backend
+ Add refs-related options to the "fsck_options", create refs-specific
+ "error_func" callback "fsck_refs_error_function".
- 1. "ref_checkee": "fsck error name": "user message".
- 2. "ref_checkee.sub_ref_checkee": "fsck error name": "user message".
- 3. "ref_checkee -> (oid hex)": "fsck error name": "user message".
- 4. "ref_checkee.sub_ref_checkee -> (oid hex)": "fsck error name": "user
- message".
-
- "fsck_refs_error_function" uses the "ref_checkee" and "sub_ref_checkee"
- in the "fsck_refs_info" to indicate the information of the checked refs.
- For loose ref and reflog, it only uses the "ref_checkee". For packed
- refs and reftable refs, when checking the consistency of the file
- itself, it still only uses "ref_checkee". However, when checking the
- consistency of the ref or reflog contained in the file ,it will use the
- "sub_ref_checkee" to indicate that we are not checking the file but the
- incorporated ref or reflog.
-
- "fsck_refs_error_function" will use the "oid" parameter if the caller
+ "fsck_refs_error_function" will use the "oid" parameter. When the caller
passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.
@@ fsck.c: int fsck_objects_error_function(struct fsck_options *o,
+int fsck_refs_error_function(struct fsck_options *options,
+ const struct object_id *oid,
+ enum object_type object_type UNUSED,
++ const struct fsck_refs_info *refs_info,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
+{
+ struct strbuf sb = STRBUF_INIT;
-+ struct fsck_refs_info *refs_info = &options->refs_info;
+ int ret = 0;
+
-+ if (the_repository->ref_storage_format == REF_STORAGE_FORMAT_FILES) {
-+ strbuf_addstr(&sb, refs_info->ref_checkee);
-+ if (refs_info->u.files.sub_ref_checkee)
-+ strbuf_addf(&sb, ".%s", refs_info->u.files.sub_ref_checkee);
++ strbuf_addstr(&sb, refs_info->path);
+
-+ if (oid)
-+ strbuf_addf(&sb, " -> (%s)", oid_to_hex(oid));
-+ }
++ if (oid)
++ strbuf_addf(&sb, " -> (%s)", oid_to_hex(oid));
+
+ if (msg_type == FSCK_WARN)
+ warning("%s: %s", sb.buf, message);
@@ fsck.h: int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *
+int fsck_refs_error_function(struct fsck_options *options,
+ const struct object_id *oid,
+ enum object_type object_type,
++ const struct fsck_refs_info *refs_info,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message);
5: c5cac2e318 ! 5: 481bffac6e refs: set up ref consistency check infrastructure
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
+static int files_fsck(struct ref_store *ref_store,
+ struct fsck_options *o)
+{
-+ int ret;
+ struct files_ref_store *refs =
+ files_downcast(ref_store, REF_STORE_READ, "fsck");
+
-+ ret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);
-+ return ret;
++ return refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);
+}
+
struct ref_storage_be refs_be_files = {
6: 84d840506e ! 6: cea8908338 git refs: add verify subcommand
@@ fsck.h: struct fsck_options {
unsigned strict:1;
+ unsigned verbose:1;
enum fsck_msg_type *msg_type;
- struct fsck_refs_info refs_info;
struct oidset skip_oids;
+ struct oidset gitmodules_found;
7: 3fc77ec329 = 7: 2200167bf6 builtin/fsck: add `git-refs verify` child process
8: 44a75141fa ! 8: c69883be9a files-backend: add unified interface for refs scanning
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
+static int files_fsck_refs(struct ref_store *ref_store,
+ struct fsck_options *o)
+{
-+ int ret;
+ files_fsck_refs_fn fsck_refs_fns[]= {
+ NULL
+ };
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
+ if (o->verbose)
+ fprintf_ln(stderr, "Checking references consistency");
+
-+ ret = files_fsck_refs_dir(ref_store, o, "refs", fsck_refs_fns);
++ return files_fsck_refs_dir(ref_store, o, "refs", fsck_refs_fns);
+
-+ return ret;
+}
+
static int files_fsck(struct ref_store *ref_store,
struct fsck_options *o)
{
-@@ refs/files-backend.c: static int files_fsck(struct ref_store *ref_store,
struct files_ref_store *refs =
files_downcast(ref_store, REF_STORE_READ, "fsck");
-- ret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);
-+ ret = refs->packed_ref_store->be->fsck(refs->packed_ref_store, o)
-+ | files_fsck_refs(ref_store, o);
-+
- return ret;
+- return refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);
++ return refs->packed_ref_store->be->fsck(refs->packed_ref_store, o) |
++ files_fsck_refs(ref_store, o);
}
+ struct ref_storage_be refs_be_files = {
9: 4a0d58b07d ! 9: 5ce0e7367e fsck: add ref name check for files backend
@@ Documentation/fsck-msgids.txt
(ERROR) A commit object has a bad parent sha1.
+`badRefName`::
-+ (ERROR) A ref has a bad name.
++ (ERROR) A ref has an invalid format.
+
`badTagName`::
(INFO) A tag has an invalid format.
@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,
+ struct dir_iterator *iter)
+{
+ struct strbuf sb = STRBUF_INIT;
++ struct fsck_refs_info info;
+ int ret = 0;
+
+ if (check_refname_format(iter->basename, REFNAME_ALLOW_ONELEVEL)) {
+ strbuf_addf(&sb, "%s/%s", refs_check_dir, iter->relative_path);
-+ o->refs_info.ref_checkee = sb.buf;
-+ ret = fsck_refs_report(o, NULL,
++ info.path = sb.buf;
++ ret = fsck_refs_report(o, NULL, &info,
+ FSCK_MSG_BAD_REF_NAME,
+ "invalid refname format");
+ }
@@ refs/files-backend.c: typedef int (*files_fsck_refs_fn)(struct fsck_options *o,
struct fsck_options *o,
const char *refs_check_dir,
@@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,
+ struct fsck_options *o)
{
- int ret;
files_fsck_refs_fn fsck_refs_fns[]= {
+ files_fsck_refs_name,
NULL
10: c529670e54 ! 10: f77ca18c68 fsck: add ref content check for files backend
@@ Documentation/fsck-msgids.txt
+ (ERROR) A ref has a bad content.
+
`badRefName`::
- (ERROR) A ref has a bad name.
+ (ERROR) A ref has an invalid format.
+`badSymrefPointee`::
+ (ERROR) The pointee of a symref is bad.
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ * be the relative path agaignst "gitdir".
+ */
+static int files_fsck_symref_target(struct fsck_options *o,
++ struct fsck_refs_info *info,
+ const char *refname,
+ const char *pointee_name,
+ const char *pointee_path)
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+
+ if (!skip_prefix(pointee_name, "refs/", &p)) {
+
-+ ret = fsck_refs_report(o, NULL,
++ ret = fsck_refs_report(o, NULL, info,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
-+ "point to target out of refs hierarchy");
++ "points to ref outside the refs directory");
+ goto out;
+ }
+
+ if (check_refname_format(pointee_name, 0)) {
-+ ret = fsck_refs_report(o, NULL,
++ ret = fsck_refs_report(o, NULL, info,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
-+ "point to invalid refname");
++ "points to refname with invalid format");
+ }
+
+ if (lstat(pointee_path, &st) < 0)
+ goto out;
+
+ if (!S_ISREG(st.st_mode) && !S_ISLNK(st.st_mode)) {
-+ ret = fsck_refs_report(o, NULL,
++ ret = fsck_refs_report(o, NULL, info,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
-+ "point to invalid target");
++ "points to an invalid file type");
+ goto out;
+ }
+out:
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ referent = STRBUF_INIT,
+ refname = STRBUF_INIT;
+ const char *trailing = NULL;
++ struct fsck_refs_info info;
+ int failure_errno = 0;
+ unsigned int type = 0;
+ struct object_id oid;
+ int ret = 0;
+
+ strbuf_addf(&refname, "%s/%s", refs_check_dir, iter->relative_path);
-+ o->refs_info.ref_checkee = refname.buf;
++ info.path = refname.buf;
+
+ /*
+ * If the file is a symlink, we need to only check the connectivity
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+
+ if (!skip_prefix(pointee_path.buf,
+ abs_gitdir.buf, &pointee_name)) {
-+ ret = fsck_refs_report(o, NULL,
++ ret = fsck_refs_report(o, NULL, &info,
+ FSCK_MSG_BAD_SYMREF_POINTEE,
+ "point to target outside gitdir");
+ goto clean;
+ }
+
-+ ret = files_fsck_symref_target(o, refname.buf, pointee_name,
-+ pointee_path.buf);
++ ret = files_fsck_symref_target(o, &info, refname.buf,
++ pointee_name, pointee_path.buf);
+ goto clean;
+ }
+
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ if (parse_loose_ref_contents(ref_content.buf, &oid,
+ &referent, &type,
+ &failure_errno, &trailing)) {
-+ ret = fsck_refs_report(o, NULL,
++ ret = fsck_refs_report(o, NULL, &info,
+ FSCK_MSG_BAD_REF_CONTENT,
+ "invalid ref content");
+ goto clean;
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
+ strbuf_addf(&pointee_path, "%s/%s", gitdir, referent.buf);
+ strbuf_rtrim(&referent);
+
-+ ret = files_fsck_symref_target(o, refname.buf, referent.buf,
-+ pointee_path.buf);
++ ret = files_fsck_symref_target(o, &info, refname.buf,
++ referent.buf, pointee_path.buf);
+ goto clean;
+ } else {
+ if (trailing && (*trailing != '\0' && *trailing != '\n')) {
-+ ret = fsck_refs_report(o, NULL,
++ ret = fsck_refs_report(o, NULL, &info,
+ FSCK_MSG_TRAILING_REF_CONTENT,
+ "trailing garbage in ref");
+ goto clean;
@@ refs/files-backend.c: static int files_fsck_refs_name(struct fsck_options *o,
struct fsck_options *o,
const char *refs_check_dir,
@@ refs/files-backend.c: static int files_fsck_refs(struct ref_store *ref_store,
- int ret;
+ {
files_fsck_refs_fn fsck_refs_fns[]= {
files_fsck_refs_name,
+ files_fsck_refs_content,
@@ t/t0602-reffiles-fsck.sh: test_expect_success 'ref name check should be adapted
+ printf "ref: refs/heads/.branch" > $branch_dir_prefix/branch-2-bad &&
+ test_must_fail git refs verify 2>err &&
+ cat >expect <<-EOF &&
-+ error: refs/heads/branch-2-bad: badSymrefPointee: point to invalid refname
++ error: refs/heads/branch-2-bad: badSymrefPointee: points to refname with invalid format
+ EOF
+ rm $branch_dir_prefix/branch-2-bad &&
+ test_cmp expect err
@@ t/t0602-reffiles-fsck.sh: test_expect_success 'ref name check should be adapted
+ printf "ref: refs/heads" > $branch_dir_prefix/branch-2-bad &&
+ test_must_fail git refs verify 2>err &&
+ cat >expect <<-EOF &&
-+ error: refs/heads/branch-2-bad: badSymrefPointee: point to invalid target
++ error: refs/heads/branch-2-bad: badSymrefPointee: points to an invalid file type
+ EOF
+ rm $branch_dir_prefix/branch-2-bad &&
+ test_cmp expect err
@@ t/t0602-reffiles-fsck.sh: test_expect_success 'ref name check should be adapted
+ printf "ref: logs/maint-v2.45" > $branch_dir_prefix/branch-2-bad &&
+ test_must_fail git refs verify 2>err &&
+ cat >expect <<-EOF &&
-+ error: refs/heads/branch-2-bad: badSymrefPointee: point to target out of refs hierarchy
++ error: refs/heads/branch-2-bad: badSymrefPointee: points to ref outside the refs directory
+ EOF
+ rm $branch_dir_prefix/branch-2-bad &&
+ test_cmp expect errSignature · 1 line
2.45.2
PATCH v13 1/10 fsck: rename "skiplist" to "skip_oids"
The "skiplist" field in "fsck_options" is related to objects. Because we are going to introduce ref consistency check, the "skiplist" name is too general which will make the caller think "skiplist" is related to both the refs and objects.
It may seem that for both refs and objects, we should provide a general "skiplist" here. However, the type for "skiplist" is struct oidset which is totally unsuitable for refs.
To avoid above ambiguity, rename "skiplist" to "skip_oids".
--- fsck.c | 4 ++-- fsck.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
fsck.c +2 -2
@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)if (!strcmp (buf,"skiplist" )) {if (equal == len)die ("skiplist requires a path" ); -oidset_parse_file (&options->skiplist, buf + equal +1 , +oidset_parse_file (&options->skip_oids, buf + equal +1 , the_repository->hash_algo); buf += len +1 ;continue ;
@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)static int object_on_skiplist (struct fsck_options *opts,const struct object_id *oid) { -return opts && oid &&oidset_contains (&opts->skiplist, oid); +return opts && oid &&oidset_contains (&opts->skip_oids, oid); } __attribute__((format (printf, 5, 6)))
fsck.h +2 -2
@@ -136,7 +136,7 @@ struct fsck_options {fsck_error error_func; unsigned strict:1;enum fsck_msg_type *msg_type; -struct oidset skiplist; +struct oidset skip_oids;struct oidset gitmodules_found;struct oidset gitmodules_done;struct oidset gitattributes_found;
@@ -145,7 +145,7 @@ struct fsck_options { };#define FSCK_OPTIONS_DEFAULT { \ - .skiplist = OIDSET_INIT, \ + .skip_oids = OIDSET_INIT, \ .gitmodules_found = OIDSET_INIT, \ .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \
Signature · 1 line
2.45.2
PATCH v13 2/10 fsck: add a unified interface for reporting fsck messages
In reply to shejialuoThe static function "report" provided by "fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs. In order to provide a unified interface which can report either objects or refs, create a new function "fsck_vreport" following the "report" prototype. Instead of using "...", provide "va_list" to allow more flexibility.
In order to provide an extensible error report for refs, add a new "fsck_refs_info" structure and add parameter "const struct *fsck_refs_info" into "fsck_vreport" function.
Like "report", the "fsck_vreport" function will use "error_func" registered in "fsck_options" to report customized messages. Change "error_func" prototype to align with the new "fsck_vreport".
Then, change "report" function to use "fsck_vreport" to report objects related messages. Add a new function called "fsck_refs_report" to use "fsck_vreport" to report refs related messages.
--- builtin/fsck.c | 1 + builtin/mktag.c | 1 + fsck.c | 56 +++++++++++++++++++++++++++++++++++++++++-------- fsck.h | 25 +++++++++++++++++++++- object-file.c | 11 +++++----- 5 files changed, 79 insertions(+), 15 deletions(-)
builtin/fsck.c +1 -0
@@ -92,6 +92,7 @@ static int objerror(struct object *obj, const char *err) static int fsck_error_func(struct fsck_options *o UNUSED,const struct object_id *oid,enum object_type object_type, + const struct fsck_refs_info *refs_info UNUSED,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message)
builtin/mktag.c +1 -0
@@ -20,6 +20,7 @@ static struct fsck_options fsck_options = FSCK_OPTIONS_STRICT; static int mktag_fsck_error_func(struct fsck_options *o UNUSED, const struct object_id *oid UNUSED, enum object_type object_type UNUSED, + const struct fsck_refs_info *refs_info UNUSED,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message)
fsck.c +47 -9
91 lines not on this page · raw message
fsck.h +24 -1
@@ -92,6 +92,7 @@ enum fsck_msg_id { };#undef MSG_ID +struct fsck_refs_info ;struct fsck_options ;struct object ;
@@ -114,23 +115,35 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);typedef int (*fsck_walk_func )(struct object *obj,enum object_type object_type,void *data,struct fsck_options *options); -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */ +/* +* callback function for reporting errors when checking either objects or refs +*/ typedef int (*fsck_error )(struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const struct fsck_refs_info *refs_info,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);int fsck_error_function (struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const struct fsck_refs_info *refs_info,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o,const struct object_id *oid,enum object_type object_type, +const struct fsck_refs_info *refs_info,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message); +/* +* The information for reporting refs-related error message +*/ +struct fsck_refs_info { +const char *path; +}; +struct fsck_options {fsck_walk_func walk;fsck_error error_func;
@@ -209,6 +222,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +/* +* Report an error or warning for refs. +*/ +__attribute__((format (printf,5 ,6 ))) +int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +const struct fsck_refs_info *refs_info, +enum fsck_msg_id msg_id, +const char *fmt, ...); +/* * Subsystem for storing human-readable names for each object. *
object-file.c +6 -5
@@ -2470,11 +2470,12 @@ int repo_has_object_file(struct repository *r,* give more context. */ static int hash_format_check_report(struct fsck_options *opts UNUSED, - const struct object_id *oid UNUSED, - enum object_type object_type UNUSED, - enum fsck_msg_type msg_type UNUSED, - enum fsck_msg_id msg_id UNUSED, -const char *message) + const struct object_id *oid UNUSED, + enum object_type object_type UNUSED, + const struct fsck_refs_info *refs_info UNUSED, + enum fsck_msg_type msg_type UNUSED, + enum fsck_msg_id msg_id UNUSED, +const char *message) {error (_ ("object fails fsck: %s" ), message);return 1 ;
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v13 02/10] fsck: add a unified interface for reporting fsck messages
Quoted 4 lines · shejialuo
The static function "report" provided by "
fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs.
Nit: it would be nice to mention _why_ it cannot be reused for refs.
Quoted 18 lines
fsck.c +6
-4@@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,return opts && oid &&oidset_contains (&opts->skip_oids, oid); } -__attribute__((format (printf,5 ,6 ))) -static int report (struct fsck_options *options, -const struct object_id *oid,enum object_type object_type, -enum fsck_msg_id msg_id,const char *fmt, ...) +/* +* Provide a unified interface for either fscking refs or objects. +* It will get the current msg error type and call the error_func callback +* which is registered in the "fsck_options" struct. +*/ +static int fsck_vreport(struct fsck_options *options,
fsck.c static int object_on_skiplist(struct fsck_options *opts, · line 243
Quoted 10 lines
fsck.c +5
-1+const struct object_id *oid, + enum object_type object_type, + const struct fsck_refs_info *refs_info, + enum fsck_msg_id msg_id, const char *fmt, va_list ap) { -va_list ap; + va_list ap_copy;struct strbuf sb =STRBUF_INIT ;enum fsck_msg_type msg_type =fsck_msg_type (msg_id, options);int result;
It is a bit weird that this new generic function receives non-generic inputs which are specific to the respective subsystems (objects or refs) that we are checking.
A better design would likely be to make error_func() receive a void pointer such that error_func() and then have the respective subsystems provide a function that knows to format the message while receiving either a struct fsck_object_report * or a struct fsck_ref_report *.
I don't think this is particularly worriesome though as it is still manageable right now. So I'm fine if we want to leave this as-is, and then we can iterate on this in a future patch series as required.
Quoted 21 lines
fsck.c +7
-3@@ -250,9 +256,9 @@ static int report(struct fsck_options *options,prepare_msg_ids ();strbuf_addf (&sb,"%s: " , msg_id_info[msg_id].camelcased); -va_start (ap, fmt); -strbuf_vaddf (&sb, fmt, ap); - result = options->error_func (options, oid, object_type, +va_copy (ap_copy, ap); +strbuf_vaddf (&sb, fmt, ap_copy); + result = options->error_func (options, oid, object_type, refs_info, msg_type, msg_id, sb.buf);strbuf_release (&sb);va_end (ap);@@ -260,6 +266,35 @@ static int report(struct fsck_options *options,return result; } +__attribute__((format (printf,5 ,6 ))) +static int report (struct fsck_options *options, +const struct object_id *oid,enum object_type object_type, +enum fsck_msg_id msg_id,const char *fmt, ...)
fsck.c static int report(struct fsck_options *options, · line 282
Quoted 10 lines
fsck.c +10
-0+{ +va_list ap; +int result; + +va_start (ap, fmt); + result =fsck_vreport (options, oid, object_type,NULL , msg_id, fmt, ap); +va_end (ap); + +return result; +}
As far as I can see, report() is now specific to reporting errors with objects while fsck_vreport() is the generic part. Do we want to rename the function to fsck_report_object() to clarify, or would that cause too much churn?
Hm. Seeing that we have 62 callsites of that function it may be too much churn indeed.
fsck.c static int report(struct fsck_options *options, · line 287
Quoted 4 lines
fsck.c +4
-0+int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +const struct fsck_refs_info *refs_info, +enum fsck_msg_id msg_id,const char *fmt, ...)
Would fsck_report_ref() be a better name?
What is the intent of the oid field? Would it be set to the object ID that a reference points to? What if the reference is a non-resolving symbolic reference? I wonder whether we can just remove it.
Patrick
Re: [GSoC][PATCH v13 02/10] fsck: add a unified interface for reporting fsck messages
Quoted 32 lines · Patrick Steinhardt
Quoted 4 lines · shejialuo
The static function "report" provided by "
fsck.c" aims at checking fsck error type and calling the callback "error_func" to report the message. However, "report" function is only related to object database which cannot be reused for refs.Nit: it would be nice to mention _why_ it cannot be reused for refs.
Quoted 24 lines
fsck.c +10
-5@@ -226,12 +226,18 @@ static int object_on_skiplist(struct fsck_options *opts,return opts && oid &&oidset_contains (&opts->skip_oids, oid); } -__attribute__((format (printf,5 ,6 ))) -static int report (struct fsck_options *options, -const struct object_id *oid,enum object_type object_type, -enum fsck_msg_id msg_id,const char *fmt, ...) +/* +* Provide a unified interface for either fscking refs or objects. +* It will get the current msg error type and call the error_func callback +* which is registered in the "fsck_options" struct. +*/ +static int fsck_vreport (struct fsck_options *options, +const struct object_id *oid, +enum object_type object_type, +const struct fsck_refs_info *refs_info, +enum fsck_msg_id msg_id,const char *fmt,va_list ap) { -va_list ap;
Quoted 9 lines
fsck.c static int object_on_skiplist(struct fsck_options *opts, · line 243
Quoted 4 lines
fsck.c +1
-0+va_list ap_copy;struct strbuf sb =STRBUF_INIT ;enum fsck_msg_type msg_type =fsck_msg_type (msg_id, options);int result;It is a bit weird that this new generic function receives non-generic inputs which are specific to the respective subsystems (objects or refs) that we are checking.
Actually, this is one of the biggest problem when implementing the infrastructure. The original function "report" only cares about reporting the problem of objects. So the callback "error_func" uses the similar prototype.
Problem comes when we want to add ref-related report. In my very former implementation, I just created a new function "fsck_refs_report" to just copy some codes from "report" and defines refs-related callback.
However, this is a bad way because we make duplication. If we want to reuse the "report" function, we should add new parameters into "report" and "error_func". This is the idea of this patch. However, as you can see, there are so many "report" function calls in the codebase, it's bad to change them. So I define a more common function called "fsck_vreport" function and wrap "report" to eventually call this function.
Quoted 5 lines
A better design would likely be to make
error_func()receive a void pointer such thaterror_func()and then have the respective subsystems provide a function that knows to format the message while receiving either astruct fsck_object_report *or astruct fsck_ref_report *.
Yes, I agree with this idea. And I think we should use only one function called "fsck_reportf" to report any fsck-related messages. We could design the following callback "prototype".
typedef int (*fsck_error)(struct fsck_options *o,
void *info,
enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message);
Thus, we could make "fsck_reportf" generic. It will handle the common "fsck_options" and "enum fsck_msg_id" and then it will call "fsck_error" callback. The user could pass either refs information or objects information.
Quoted 4 lines
I don't think this is particularly worriesome though as it is still manageable right now. So I'm fine if we want to leave this as-is, and then we can iterate on this in a future patch series as required.
I strongly suggest that we should use the above design for the following reasons:
1. We only expose one interface called "fsck_reportf" which will make the code clear. Actually, there is no different between reporting refs and reporting objects.
2. We provide more extensibility here, because we will never change "fsck_reportf" and "fsck_error" prototype when we want to add more info for either refs or objects.
But do we really need this? Junio, could you please give some advice here. How do you think about this design. In my perspective, the only overhead here is that there are too many "report" function we should refactor.
Quoted 37 lines
Quoted 21 lines
fsck.c +7
-3@@ -250,9 +256,9 @@ static int report(struct fsck_options *options,prepare_msg_ids ();strbuf_addf (&sb,"%s: " , msg_id_info[msg_id].camelcased); -va_start (ap, fmt); -strbuf_vaddf (&sb, fmt, ap); - result = options->error_func (options, oid, object_type, +va_copy (ap_copy, ap); +strbuf_vaddf (&sb, fmt, ap_copy); + result = options->error_func (options, oid, object_type, refs_info, msg_type, msg_id, sb.buf);strbuf_release (&sb);va_end (ap);@@ -260,6 +266,35 @@ static int report(struct fsck_options *options,return result; } +__attribute__((format (printf,5 ,6 ))) +static int report (struct fsck_options *options, +const struct object_id *oid,enum object_type object_type, +enum fsck_msg_id msg_id,const char *fmt, ...)fsck.c static int report(struct fsck_options *options, · line 282
Quoted 10 lines
fsck.c +10
-0+{ +va_list ap; +int result; + +va_start (ap, fmt); + result =fsck_vreport (options, oid, object_type,NULL , msg_id, fmt, ap); +va_end (ap); + +return result; +}As far as I can see,
report()is now specific to reporting errors with objects whilefsck_vreport()is the generic part. Do we want to rename the function tofsck_report_object()to clarify, or would that cause too much churn?
Quoted 3 lines
Hm. Seeing that we have 62 callsites of that function it may be too much churn indeed.
Yes, there are too many references for "report" function. That's why I wrap the "report" using "fsck_vreport".
Quoted 7 lines
fsck.c static int report(struct fsck_options *options, · line 287
Quoted 4 lines
fsck.c +4
-0+int fsck_refs_report (struct fsck_options *options, +const struct object_id *oid, +const struct fsck_refs_info *refs_info, +enum fsck_msg_id msg_id,const char *fmt, ...)Would
fsck_report_ref()be a better name?
I agree. However, if we use the above design, we will just use "fsck_reportf" here both for refs and objects.
Quoted 4 lines
What is the intent of the
oidfield? Would it be set to the object ID that a reference points to? What if the reference is a non-resolving symbolic reference? I wonder whether we can just remove it.
oid is used to be the object ID that a reference points to. If the reference is a symbolic link or symref, we do not care about it. The caller should just pass NULL. Actually, we may not use this field. I just suppose that we may provide the user more information. Because when using "file-backend.c::parse_loose_ref_contents()" we will automatically get the oid if the ref is a regular reference. So I just provide oid here.
Quoted 1 line
Patrick
PATCH v13 3/10 fsck: rename objects-related fsck error functions
In reply to shejialuoThe names of objects-related fsck error functions are general. It's OK when there is only object database check. However, we have introduced refs database check report function. To avoid ambiguity, rename object-related fsck error functions to explicitly indicate these functions are used to report objects-related messages.
--- builtin/fsck.c | 16 ++++++++-------- fsck.c | 32 ++++++++++++++++---------------- fsck.h | 30 +++++++++++++++--------------- 3 files changed, 39 insertions(+), 39 deletions(-)
builtin/fsck.c +8 -8
@@ -89,13 +89,13 @@ static int objerror(struct object *obj, const char *err)return -1 ; } -static int fsck_error_func(struct fsck_options *o UNUSED, -const struct object_id *oid, -enum object_type object_type, - const struct fsck_refs_info *refs_info UNUSED, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +static int fsck_objects_error_func(struct fsck_options *o UNUSED, +const struct object_id *oid, +enum object_type object_type, + const struct fsck_refs_info *refs_info UNUSED, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {switch (msg_type) {case FSCK_WARN :
@@ -939,7 +939,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) fsck_walk_options.walk = mark_object; fsck_obj_options.walk = mark_used; - fsck_obj_options.error_func = fsck_error_func; + fsck_obj_options.error_func = fsck_objects_error_func;if (check_strict) fsck_obj_options.strict =1 ;
fsck.c +16 -16
@@ -1235,13 +1235,13 @@ int fsck_buffer(const struct object_id *oid, enum object_type type, type); } -int fsck_error_function (struct fsck_options *o, -const struct object_id *oid, - enum object_type object_type UNUSED, - const struct fsck_refs_info *refs_info UNUSED, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +int fsck_objects_error_function (struct fsck_options *o, +const struct object_id *oid, + enum object_type object_type UNUSED, + const struct fsck_refs_info *refs_info UNUSED, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {if (msg_type ==FSCK_WARN ) {warning ("object %s: %s" ,fsck_describe_object (o, oid), message);
@@ -1339,18 +1339,18 @@ int git_fsck_config(const char *var, const char *value,* Custom error callbacks that are used in more than one place. */ -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, -const struct fsck_refs_info *refs_info, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message) +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +const struct object_id *oid, +enum object_type object_type, +const struct fsck_refs_info *refs_info, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message) {if (msg_id ==FSCK_MSG_GITMODULES_MISSING ) {puts (oid_to_hex (oid));return 0 ; } -return fsck_error_function (o, oid, object_type, refs_info, - msg_type, msg_id, message); +return fsck_objects_error_function (o, oid, object_type, refs_info, + msg_type, msg_id, message); }
fsck.h +15 -15
@@ -124,18 +124,18 @@ typedef int (*fsck_error)(struct fsck_options *o, enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); -int fsck_error_function (struct fsck_options *o, -const struct object_id *oid,enum object_type object_type, -const struct fsck_refs_info *refs_info, -enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, -const char *message); -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, -const struct fsck_refs_info *refs_info, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message); +int fsck_objects_error_function (struct fsck_options *o, +const struct object_id *oid,enum object_type object_type, +const struct fsck_refs_info *refs_info, +enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, +const char *message); +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +const struct object_id *oid, +enum object_type object_type, +const struct fsck_refs_info *refs_info, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);/* * The information for reporting refs-related error message
@@ -163,7 +163,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function \ + .error_func = fsck_objects_error_function \ }#define FSCK_OPTIONS_STRICT { \ .strict = 1, \
@@ -171,7 +171,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function, \ + .error_func = fsck_objects_error_function, \ }#define FSCK_OPTIONS_MISSING_GITMODULES { \ .strict = 1, \
@@ -179,7 +179,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_cb_print_missing_gitmodules, \ + .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ }/* descend in all linked child objects
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v13 03/10] fsck: rename objects-related fsck error functions
Quoted 1 line · shejialuo
The names of objects-related fsck error functions are general. It's OK
s/general/generic, I guess.
Quoted 4 lines
when there is only object database check. However, we have introduced refs database check report function. To avoid ambiguity, rename object-related fsck error functions to explicitly indicate these functions are used to report objects-related messages.
I agree that it is sensible to rename the functions to make it more explicit that they are only used by the object-checking infrastructure.
Patrick
Re: [GSoC][PATCH v13 03/10] fsck: rename objects-related fsck error functions
Quoted 5 lines · Patrick Steinhardt
Quoted 1 line · shejialuo
The names of objects-related fsck error functions are general. It's OK
s/general/generic, I guess.
I will improve this in the next version.
PATCH v13 4/10 fsck: add refs-related error report function
In reply to shejialuoAdd refs-related options to the "fsck_options", create refs-specific "error_func" callback "fsck_refs_error_function".
"fsck_refs_error_function" will use the "oid" parameter. When the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.
Last, add "FSCK_REFS_OPTIONS_DEFAULT" and "FSCK_REFS_OPTIONS_STRICT" macros to create refs options easily.
--- fsck.c | 25 +++++++++++++++++++++++++ fsck.h | 14 ++++++++++++++ 2 files changed, 39 insertions(+)
fsck.c +25 -0
@@ -1251,6 +1251,31 @@ int fsck_objects_error_function(struct fsck_options *o,return 1 ; } +int fsck_refs_error_function (struct fsck_options *options, +const struct object_id *oid, + enum object_type object_type UNUSED, +const struct fsck_refs_info *refs_info, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) +{ +struct strbuf sb =STRBUF_INIT ; +int ret =0 ; + +strbuf_addstr (&sb, refs_info->path); + +if (oid) +strbuf_addf (&sb," -> (%s)" ,oid_to_hex (oid)); + +if (msg_type ==FSCK_WARN ) +warning ("%s: %s" , sb.buf, message); +else + ret =error ("%s: %s" , sb.buf, message); + +strbuf_release (&sb); +return ret; +} +static int fsck_blobs (struct oidset *blobs_found,struct oidset *blobs_done,enum fsck_msg_id msg_missing,enum fsck_msg_id msg_type,struct fsck_options *options,const char *blob_type)
fsck.h +14 -0
@@ -136,6 +136,13 @@ int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); +int fsck_refs_error_function (struct fsck_options *options, +const struct object_id *oid, +enum object_type object_type, +const struct fsck_refs_info *refs_info, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message);/* * The information for reporting refs-related error message
@@ -181,6 +188,13 @@ struct fsck_options { .gitattributes_done = OIDSET_INIT, \ .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ } +#define FSCK_REFS_OPTIONS_DEFAULT { \ + .error_func = fsck_refs_error_function, \ +} +#define FSCK_REFS_OPTIONS_STRICT { \ + .strict = 1, \ + .error_func = fsck_refs_error_function, \ +}/* descend in all linked child objects * the return value is:
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v13 04/10] fsck: add refs-related error report function
Quoted 2 lines · shejialuo
Add refs-related options to the "
fsck_options", create refs-specific "error_func" callback "fsck_refs_error_function".
We should have an explanation _why_ we are adding these functions in the commit message.
Quoted 6 lines
"
fsck_refs_error_function" will use the "oid" parameter. When the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.Last, add "
FSCK_REFS_OPTIONS_DEFAULT" and "FSCK_REFS_OPTIONS_STRICT" macros to create refs options easily.
It is a bit unclear to me what you mean with "create refs options easily". Do you mean to say that git refs check (or whatever this will be called) will have flags like "--strict"?
Quoted 21 lines
Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- fsck.c | 25 +++++++++++++++++++++++++ fsck.h | 14 ++++++++++++++ 2 files changed, 39 insertions(+)fsck.c +5
-0@@ -1251,6 +1251,31 @@ int fsck_objects_error_function(struct fsck_options *o,return 1 ; } +int fsck_refs_error_function(struct fsck_options *options, + const struct object_id *oid, + enum object_type object_type UNUSED, + const struct fsck_refs_info *refs_info, + enum fsck_msg_type msg_type,
fsck.c int fsck_objects_error_function(struct fsck_options *o, · line 1268
Quoted 10 lines
fsck.c +10
-0+ enum fsck_msg_id msg_id UNUSED, + const char *message) +{ + struct strbuf sb = STRBUF_INIT; +int ret =0 ; + +strbuf_addstr (&sb, refs_info->path); + +if (oid) +strbuf_addf (&sb," -> (%s)" ,oid_to_hex (oid));
Okay, so we do end up printing the object ID indeed. But wouldn't we want to potentially do the same with symbolic refs?
Also, would it make more sense to put the oid (and potentially the referent when we also handle symbolic refs) into struct fsck_refs_info? Like this, the whole state would be self-contained in that structure, which would also make my proposal from a preceding commit more feasible where the subsystem-specific error functions only get a void pointer to this structure. It would require another refactoring on top to move the object type and OID into a struct fsck_objects_info, too, but that shouldn't be too involved, I guess.
Patrick
Re: [GSoC][PATCH v13 04/10] fsck: add refs-related error report function
Quoted 7 lines · Patrick Steinhardt
Quoted 2 lines · shejialuo
Add refs-related options to the "
fsck_options", create refs-specific "error_func" callback "fsck_refs_error_function".We should have an explanation _why_ we are adding these functions in the commit message.
Yes, I will improve this in the next version.
Quoted 11 lines
Quoted 6 lines
"
fsck_refs_error_function" will use the "oid" parameter. When the caller passes the oid, it will use "oid_to_hex" to get the corresponding hex value to report to the caller.Last, add "
FSCK_REFS_OPTIONS_DEFAULT" and "FSCK_REFS_OPTIONS_STRICT" macros to create refs options easily.It is a bit unclear to me what you mean with "create refs options easily". Do you mean to say that
git refs check(or whatever this will be called) will have flags like "--strict"?
Yes, when the user passes --strict, all the warn type will be seen as the error type. So I create "FSCK_REFS_OPTIONS_STRICT". However, I didn't think too much here. I just followed the way the codebase does for the objects.
Quoted 35 lines
Quoted 21 lines
Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- fsck.c | 25 +++++++++++++++++++++++++ fsck.h | 14 ++++++++++++++ 2 files changed, 39 insertions(+)fsck.c +5
-0@@ -1251,6 +1251,31 @@ int fsck_objects_error_function(struct fsck_options *o,return 1 ; } +int fsck_refs_error_function(struct fsck_options *options, + const struct object_id *oid, + enum object_type object_type UNUSED, + const struct fsck_refs_info *refs_info, + enum fsck_msg_type msg_type,fsck.c int fsck_objects_error_function(struct fsck_options *o, · line 1268
Quoted 10 lines
fsck.c +10
-0+ enum fsck_msg_id msg_id UNUSED, + const char *message) +{ + struct strbuf sb = STRBUF_INIT; +int ret =0 ; + +strbuf_addstr (&sb, refs_info->path); + +if (oid) +strbuf_addf (&sb," -> (%s)" ,oid_to_hex (oid));Okay, so we do end up printing the object ID indeed. But wouldn't we want to potentially do the same with symbolic refs?
Quoted 9 lines
Also, would it make more sense to put the
oid(and potentially thereferentwhen we also handle symbolic refs) intostruct fsck_refs_info? Like this, the whole state would be self-contained in that structure, which would also make my proposal from a preceding commit more feasible where the subsystem-specific error functions only get a void pointer to this structure. It would require another refactoring on top to move the object type and OID into astruct fsck_objects_info, too, but that shouldn't be too involved, I guess.
Yes, I totally agree here. I didn't consider symrefs here, actually we should provide "symref -> referent" message if we have providen the "regular ref -> oid" message.
I think we should do this. And I have commented on "[Patch v13][2/10]", it is a necessity we should refactor this part.
Quoted 1 line
Patrick
PATCH v13 5/10 refs: set up ref consistency check infrastructure
In reply to shejialuoThe interfaces defined in the ref_storage_be are carefully structured in semantic. It's organized as the five parts:
1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.
To keep consistent with the git-fsck(1), add a new interface named "fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.
Last, implement placeholder functions for each ref backends.
--- refs.c | 5 +++++ refs.h | 8 ++++++++ refs/debug.c | 11 +++++++++++ refs/files-backend.c | 13 ++++++++++++- refs/packed-backend.c | 8 ++++++++ refs/refs-internal.h | 6 ++++++ refs/reftable-backend.c | 8 ++++++++ 7 files changed, 58 insertions(+), 1 deletion(-)
refs.c +5 -0
@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)return check_or_sanitize_refname (refname, flags,NULL ); } +int refs_fsck (struct ref_store *refs,struct fsck_options *o) +{ +return refs->be->fsck (refs, o); +} +void sanitize_refname_component (const char *refname,struct strbuf *out) {if (check_or_sanitize_refname (refname,REFNAME_ALLOW_ONELEVEL , out))
refs.h +8 -0
@@ -4,6 +4,7 @@#include "commit.h" #include "repository.h" +struct fsck_options ;struct object_id ;struct ref_store ;struct strbuf ;
@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat*/ int check_refname_format (const char *refname,int flags); +/* +* Check the reference database for consistency. Return 0 if refs and +* reflogs are consistent, and non-zero otherwise. The errors will be +* written to stderr. +*/ +int refs_fsck (struct ref_store *refs,struct fsck_options *o); +/* * Apply the rules from check_refname_format, but mutate the result until it * is acceptable, and place the result in "out".
refs/debug.c +11 -0
@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,return res; } +static int debug_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store; +int res = drefs->refs->be->fsck (drefs->refs, o); +trace_printf_key (&trace_refs,"fsck: %d\n" , res); +return res; +} +struct ref_storage_be refs_be_debug = { .name ="debug" , .init =NULL ,
@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = { .create_reflog = debug_create_reflog, .delete_reflog = debug_delete_reflog, .reflog_expire = debug_reflog_expire, + + .fsck = debug_fsck, };
refs/files-backend.c +12 -1
@@ -3408,6 +3408,15 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +static int files_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +struct files_ref_store *refs = +files_downcast (ref_store,REF_STORE_READ ,"fsck" ); + +return refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); +} +struct ref_storage_be refs_be_files = { .name ="files" , .init = files_ref_store_init,
@@ -3434,5 +3443,7 @@ struct ref_storage_be refs_be_files = { .reflog_exists = files_reflog_exists, .create_reflog = files_create_reflog, .delete_reflog = files_delete_reflog, - .reflog_expire = files_reflog_expire + .reflog_expire = files_reflog_expire, + + .fsck = files_fsck, };
refs/packed-backend.c +8 -0
@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_sreturn empty_ref_iterator_begin (); } +static int packed_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_packed = { .name ="packed" , .init = packed_ref_store_init,
@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = { .create_reflog = NULL, .delete_reflog = NULL, .reflog_expire = NULL, + + .fsck = packed_fsck, };
refs/refs-internal.h +6 -0
@@ -4,6 +4,7 @@#include "refs.h" #include "iterator.h" +struct fsck_options ;struct ref_transaction ;/*
@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,typedef int read_symbolic_ref_fn (struct ref_store *ref_store,const char *refname,struct strbuf *referent); +typedef int fsck_fn (struct ref_store *ref_store, +struct fsck_options *o); +struct ref_storage_be {const char *name;ref_store_init_fn *init;
@@ -677,6 +681,8 @@ struct ref_storage_be {create_reflog_fn *create_reflog;delete_reflog_fn *delete_reflog;reflog_expire_fn *reflog_expire; + +fsck_fn *fsck; };extern struct ref_storage_be refs_be_files;
refs/reftable-backend.c +8 -0
@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,return ret; } +static int reftable_be_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_reftable = { .name ="reftable" , .init = reftable_be_init,
@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = { .create_reflog = reftable_be_create_reflog, .delete_reflog = reftable_be_delete_reflog, .reflog_expire = reftable_be_reflog_expire, + + .fsck = reftable_be_fsck, };
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v13 05/10] refs: set up ref consistency check infrastructure
Quoted 9 lines · shejialuo
The interfaces defined in the
ref_storage_beare carefully structured in semantic. It's organized as the five parts:1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.
Quoted 6 lines
To keep consistent with the git-fsck(1), add a new interface named "
fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.Last, implement placeholder functions for each ref backends.
You're carefully explaining what you are doing and where you are placing the new callback functions. But you never explain why you add those functions in the first place, which I would think is much more important than explaining the placement of the new callbacks.
Other than that this patch looks good to me.
Patrick
Re: [GSoC][PATCH v13 05/10] refs: set up ref consistency check infrastructure
Quoted 15 lines · Patrick Steinhardt
Quoted 14 lines · shejialuo
The interfaces defined in the
ref_storage_beare carefully structured in semantic. It's organized as the five parts:1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.To keep consistent with the git-fsck(1), add a new interface named "
fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.
Quoted 7 lines
Quoted 1 line
Last, implement placeholder functions for each ref backends.
You're carefully explaining what you are doing and where you are placing the new callback functions. But you never explain why you add those functions in the first place, which I would think is much more important than explaining the placement of the new callbacks.
Thanks, I will improve this in the next version.
Quoted 3 lines
Other than that this patch looks good to me.
Patrick
PATCH v13 6/10 git refs: add verify subcommand
In reply to shejialuoIntroduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency and also this subcommand will be used as the entry point of checking refs for "git-fsck(1)". Last, add "verbose" field into "fsck_options" to indicate whether we should print verbose messages when checking refs and objects consistency.
--- Documentation/git-refs.txt | 13 +++++++++++ builtin/refs.c | 44 ++++++++++++++++++++++++++++++++++++++ fsck.h | 1 + 3 files changed, 58 insertions(+)
Documentation/git-refs.txt +13 -0
@@ -10,6 +10,7 @@ SYNOPSIS -------- [verse] 'git refs migrate' --ref-format=<format> [--dry-run] +'git refs verify' [--strict] [--verbose] DESCRIPTION -----------
@@ -22,6 +23,9 @@ COMMANDS migrate:: Migrate ref store between different formats. +verify:: + Verify reference database consistency. + OPTIONS -------
@@ -39,6 +43,15 @@ include::ref-storage-format.txt[] can be used to double check that the migration works as expected before performing the actual migration. +The following options are specific to 'git refs verify': + +--strict:: + Enable more strict checking, every WARN severity for the `Fsck Messages` + be seen as ERROR. See linkgit:git-fsck[1]. + +--verbose:: + When verifying the reference database consistency, be chatty. + KNOWN LIMITATIONS -----------------
builtin/refs.c +44 -0
69 lines not on this page · raw message
fsck.h +1 -0
@@ -155,6 +155,7 @@ struct fsck_options {fsck_walk_func walk;fsck_error error_func; unsigned strict:1; + unsigned verbose:1;enum fsck_msg_type *msg_type;struct oidset skip_oids;struct oidset gitmodules_found;
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v13 06/10] git refs: add verify subcommand
On Mon, Jul 29, 2024 at 09:27:12PM +0800, shejialuo wrote:
The subject should probably start with "builtin/refs", not "git refs".
Quoted 5 lines
Introduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency and also this subcommand will be used as the entry point of checking refs for "git-fsck(1)". Last, add "verbose" field into "
fsck_options" to indicate whether we should print verbose messages when checking refs and objects consistency.
Nice. I very much like that we now have a common home for such low-level ref-related commands. Also, "verify" is neatly in line with e.g. `git commit-graph verify".
Documentation/git-refs.txt include::ref-storage-format.txt[] · line 50
Quoted 9 lines
Documentation/git-refs.txt +5
-0@@ -39,6 +43,15 @@ include::ref-storage-format.txt[] can be used to double check that the migration works as expected before performing the actual migration. +The following options are specific to 'git refs verify': + +--strict:: + Enable more strict checking, every WARN severity for the `Fsck Messages` + be seen as ERROR. See linkgit:git-fsck[1].
How about:
"Enable stricter error checking. This will cause warnings to be reported as errors. See linkgit:git-fsck[1]."
Documentation/git-refs.txt include::ref-storage-format.txt[] · line 53
Quoted 2 lines
Documentation/git-refs.txt +2
-0+--verbose:: + When verifying the reference database consistency, be chatty.
I wonder whether this really helps all that much. It doesn't really say what it adds on top of the default mode. So unless we document what exactly this changes, I rather think we can just leave it aways as basically everyone knows what a "--verbose" flag does.
Quoted 25 lines
Documentation/git-refs.txt +0
-0KNOWN LIMITATIONS -----------------builtin/refs.c +5
-0@@ -1,4 +1,6 @@#include "builtin.h" +#include "config.h" +#include "fsck.h" #include "parse-options.h" #include "refs.h" #include "repository.h" @@ -7,6 +9,9 @@#define REFS_MIGRATE_USAGE \ N_("git refs migrate --ref-format=<format> [--dry-run]") +#define REFS_VERIFY_USAGE \ + N_("git refs verify [--strict] [--verbose]") +static int cmd_refs_migrate (int argc,const char **argv,const char *prefix) {const char *const migrate_usage[] = {
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 68
Quoted 6 lines
builtin/refs.c +3
-0@@ -58,15 +63,54 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)return err; } +static int cmd_refs_verify (int argc,const char **argv,const char *prefix) +{ +struct fsck_options fsck_refs_options =FSCK_REFS_OPTIONS_DEFAULT ;
So we don't ever end up using FSCK_REFS_OPTIONS_STRICT? If so, I think we should just drop that declaration in the preceding patch.
Quoted 10 lines
builtin/refs.c +10
-0+const char *const verify_usage[] = { +REFS_VERIFY_USAGE , +NULL , + }; +unsigned int verbose =0 , strict =0 ; +struct option options[] = { +OPT__VERBOSE (&verbose,N_ ("be verbose" )), +OPT_BOOL (0 ,"strict" , &strict,N_ ("enable strict checking" )), +OPT_END (), + };
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 88
Quoted 10 lines
builtin/refs.c +10
-0+int ret; + + argc =parse_options (argc, argv, prefix, options, verify_usage,0 ); +if (argc) +usage (_ ("'git refs verify' takes no arguments" )); + +if (verbose) + fsck_refs_options.verbose =1 ; +if (strict) + fsck_refs_options.strict =1 ;
Instead of manually setting those variables, we can pass pointers to those member variables in the struct options directly.
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 99
Quoted 10 lines
builtin/refs.c +10
-0+git_config (git_fsck_config, &fsck_refs_options); +prepare_repo_settings (the_repository); + + ret =refs_fsck (get_main_ref_store (the_repository), &fsck_refs_options); + +/* +* Explicitly free the allocated array and "skip_oids" set +*/ +free (fsck_refs_options.msg_type); +oidset_clear (&fsck_refs_options.skip_oids);
Should we provide a fsck_options_clear() function that does this for us? Otherwise we'll have to adapt callsites of refs_fsck whenever internal implementation details of the subsystem add newly allocated members.
Quoted 22 lines
builtin/refs.c +5
-0+return ret; +} +int cmd_refs (int argc,const char **argv,const char *prefix) {const char *const refs_usage[] = {REFS_MIGRATE_USAGE , +REFS_VERIFY_USAGE ,NULL , };parse_opt_subcommand_fn *fn =NULL ;struct option opts[] = {OPT_SUBCOMMAND ("migrate" , &fn, cmd_refs_migrate), +OPT_SUBCOMMAND ("verify" , &fn, cmd_refs_verify),OPT_END (), };fsck.h +0
-0
fsck.h struct fsck_options { · line 158
Quoted 4 lines
fsck.h +1
-0@@ -155,6 +155,7 @@ struct fsck_options {fsck_walk_func walk;fsck_error error_func; unsigned strict:1; + unsigned verbose:1;
Okay. Let's see whether this field will be used in a subsequent patch. If not, we should drop it and get rid of the option altogether, I guess.
Patrick
Re: [GSoC][PATCH v13 06/10] git refs: add verify subcommand
Quoted 4 lines · Patrick Steinhardt
On Mon, Jul 29, 2024 at 09:27:12PM +0800, shejialuo wrote:
The subject should probably start with "builtin/refs", not "git refs".
Yes, I will improve this in the next version.
Quoted 12 lines
Documentation/git-refs.txt include::ref-storage-format.txt[] · line 50
Quoted 9 lines
Documentation/git-refs.txt +5
-0@@ -39,6 +43,15 @@ include::ref-storage-format.txt[] can be used to double check that the migration works as expected before performing the actual migration. +The following options are specific to 'git refs verify': + +--strict:: + Enable more strict checking, every WARN severity for the `Fsck Messages` + be seen as ERROR. See linkgit:git-fsck[1].How about:
Quoted 3 lines
"Enable stricter error checking. This will cause warnings to be reported as errors. See linkgit:git-fsck[1]."
Yes, it is much more clear. Actually, I really feel hard to write a good document.
Quoted 8 lines
Documentation/git-refs.txt include::ref-storage-format.txt[] · line 53
Quoted 2 lines
Documentation/git-refs.txt +2
-0+--verbose:: + When verifying the reference database consistency, be chatty.I wonder whether this really helps all that much. It doesn't really say what it adds on top of the default mode. So unless we document what exactly this changes, I rather think we can just leave it aways as basically everyone knows what a "--verbose" flag does.
Yes, I think so. --verbose is a common flag. However, we have already added this, so we may just leave it here. It's not bad to add more information.
Quoted 7 lines
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 68
Quoted 3 lines
builtin/refs.c +3
-0+static int cmd_refs_verify (int argc,const char **argv,const char *prefix) +{ +struct fsck_options fsck_refs_options =FSCK_REFS_OPTIONS_DEFAULT ;So we don't ever end up using
FSCK_REFS_OPTIONS_STRICT? If so, I think we should just drop that declaration in the preceding patch.
I agree here. I will delete FSCK_REFS_OPTIONS_STRICT.
Quoted 15 lines
Quoted 15 lines
builtin/refs.c +15
-0+const char *const verify_usage[] = { +REFS_VERIFY_USAGE , +NULL , + }; +unsigned int verbose =0 , strict =0 ; +struct option options[] = { +OPT__VERBOSE (&verbose,N_ ("be verbose" )), +OPT_BOOL (0 ,"strict" , &strict,N_ ("enable strict checking" )), +OPT_END (), + }; +int ret; + + argc =parse_options (argc, argv, prefix, options, verify_usage,0 ); +if (argc) +usage (_ ("'git refs verify' takes no arguments" ));
Quoted 9 lines
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 88
Quoted 5 lines
builtin/refs.c +5
-0+ +if (verbose) + fsck_refs_options.verbose =1 ; +if (strict) + fsck_refs_options.strict =1 ;Instead of manually setting those variables, we can pass pointers to those member variables in the
struct options directly.
Yes, but I have tried but found that the types are mismatching, I will find a way to do this.
Quoted 7 lines
Quoted 7 lines
builtin/refs.c +7
-0+git_config (git_fsck_config, &fsck_refs_options); +prepare_repo_settings (the_repository); + + ret =refs_fsck (get_main_ref_store (the_repository), &fsck_refs_options); + +/* +* Explicitly free the allocated array and "skip_oids" set
Quoted 9 lines
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 99
Quoted 3 lines
builtin/refs.c +3
-0+*/ +free (fsck_refs_options.msg_type); +oidset_clear (&fsck_refs_options.skip_oids);Should we provide a
fsck_options_clear()function that does this for us? Otherwise we'll have to adapt callsites ofrefs_fsckwhenever internal implementation details of the subsystem add newly allocated members.
Yes, I agree with this. I wanna talk more on this. In the first time, I did not call oidset_clear and I failed the CI tests. It made me confused. Because we never use "skip_oids" in the ref check, why the tests said that "fsck_refs.options.skip_oids" was not freed.
This is because when executing the command "git -c fsck.skipList=.. fsck", in the subprocess git refs verify, the code will still setup the "skip_oids" by the config. So we should explicitly free the "skip_oids".
But how does "fsck.c" free "skip_oids", actually "fsck.c" never frees "skip_oids". This is because "git-fsck(1)" defines the following:
static struct fsck_options fsck_walk_options = FSCK_OPTIONS_DEFAULT; static struct fsck_options fsck_obj_options = FSCK_OPTIONS_DEFAULT;
Because these two options are "static", so there is no memory leak. We leave it to the operating system. So maybe a more simple way is just to add "static" identifier in "cmd_refs_verify" which means:
- struct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;
+ static struct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;
But I don't think we should use static, because Eric has told me that making a variable "static" will make the code harder to "libfy". So let's use "fsck_options_clear" function instead.
Re: [GSoC][PATCH v13 06/10] git refs: add verify subcommand
Quoted 23 lines · shejialuo
Quoted 11 lines · Patrick Steinhardt
builtin/refs.c static int cmd_refs_migrate(int argc, const char **argv, const char *prefix) · line 99
Quoted 5 lines · shejialuo
builtin/refs.c +5
-0+/* +* Explicitly free the allocated array and "skip_oids" set +*/ +free (fsck_refs_options.msg_type); +oidset_clear (&fsck_refs_options.skip_oids);Should we provide a
fsck_options_clear()function that does this for us? Otherwise we'll have to adapt callsites ofrefs_fsckwhenever internal implementation details of the subsystem add newly allocated members.[...]
But how does "fsck.c" free "skip_oids", actually "fsck.c" never frees "skip_oids". This is because "git-fsck(1)" defines the following:static struct fsck_options fsck_walk_options = FSCK_OPTIONS_DEFAULT; static struct fsck_options fsck_obj_options = FSCK_OPTIONS_DEFAULT;Because these two options are "static", so there is no memory leak. We leave it to the operating system. So maybe a more simple way is just to add "static" identifier in "
cmd_refs_verify" which means:
Quoted 6 lines
- struct
fsck_optionsfsck_refs_options=FSCK_REFS_OPTIONS_DEFAULT;
+ static structfsck_optionsfsck_refs_options=FSCK_REFS_OPTIONS_DEFAULT;But I don't think we should use
static, because Eric has told me that making a variable "static" will make the code harder to "libfy". So let's use "fsck_options_clear" function instead.
I haven't been following this topic closely and I'm not familiar with this code (and don't have much time now to dig into it), but I suspect the context here is rather different from the one[*] in which I was highly skeptical of the use of static. The static in that earlier case was suspicious/questionable for two reasons. First, it was a case of premature optimization (which, by definition, is frowned upon). Second, it was in a "library" function (namely, top-level fsck.c:fsck_refs_error_function()) which may someday become a linkable library which other programs (aside from git itself) may utilize. Having a static strbuf in the library function makes the function non-reentrant and takes memory management out of the hands of the client.
In the case under discussion here (namely builtin/fsck.c), it is a Git-specific command, not library code. As such, "libification" is much less of an issue since Git-specific command code is less likely to be reused by some other project. (However, that's not to say that we shouldn't worry about unnecessary use of static even in builtin commands; code from those commands does periodically migrate from builtin/*.c to top-level library oriented *.c.)
So, considering that the variable under discussion:
struct fsck_options fsck_refs_options = FSCK_REFS_OPTIONS_DEFAULT;
is part of a builtin command rather than library code, we don't have to worry about "libification" so much, thus making it static would be a workable approach. However, doing so merely to avoid complaint by the leak-checker does not seem like good justification. Hence, keeping this variable non-static and freeing it explicitly seems a better idea (which is what this code does presently).
I do agree with Patrick that adding fsck_options_clear() to top-level fsck.h would be sensible since it frees callers from having to know implementation details of fsck_options.
By the way, regarding the static fsck_walk_options and fsck_obj_options those are probably global static for convenience rather than out of necessity. It might very well be possible to make those local variables in builtin/fsck.c:cmd_fsck() and then plumb them through to called functions so that they don't have to be static, and then they would be freed manually by cmd_fsck(), as well. However, that sort of change is well outside the scope of this topic.
[*] https://lore.kernel.org/git/CAPig+cR=RgMeaAy1PRGgHu6_Ak+7=_-5tGvBZRekKRxi7GtdHw@mail.gmail.com/
PATCH v13 7/10 builtin/fsck: add `git-refs verify` child process
In reply to shejialuoIntroduce a new function "fsck_refs" that initializes and runs a child process to execute the "git-refs verify" command.
--- builtin/fsck.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
builtin/fsck.c +17 -0
@@ -897,6 +897,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)return res; } +static void fsck_refs (void ) +{ +struct child_process refs_verify =CHILD_PROCESS_INIT ; +child_process_init (&refs_verify); + refs_verify.git_cmd =1 ; +strvec_pushl (&refs_verify.args,"refs" ,"verify" ,NULL ); +if (verbose) +strvec_push (&refs_verify.args,"--verbose" ); +if (check_strict) +strvec_push (&refs_verify.args,"--strict" ); + +if (run_command (&refs_verify)) + errors_found |=ERROR_REFS ; +} +static char const *const fsck_usage[] = { N_("git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\n" " [--[no-]full] [--strict] [--verbose] [--lost-found]\n"
@@ -1066,6 +1081,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)check_connectivity (); +fsck_refs (); +if (the_repository->settings.core_commit_graph) {struct child_process commit_graph_verify =CHILD_PROCESS_INIT ;
Signature · 1 line
2.45.2
PATCH v13 8/10 files-backend: add unified interface for refs scanning
In reply to shejialuoFor refs and reflogs, we need to scan its corresponding directories to check every regular file or symbolic link which shares the same pattern. Introduce a unified interface for scanning directories for files-backend.
--- refs/files-backend.c | 74 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 1 deletion(-)
refs/files-backend.c +73 -1
@@ -6,6 +6,7 @@#include "../gettext.h" #include "../hash.h" #include "../hex.h" +#include "../fsck.h" #include "../refs.h" #include "refs-internal.h" #include "ref-cache.h"
@@ -3408,13 +3409,84 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +/* +* For refs and reflogs, they share a unified interface when scanning +* the whole directory. This function is used as the callback for each +* regular file or symlink in the directory. +*/ +typedef int (*files_fsck_refs_fn )(struct fsck_options *o, +const char *gitdir, +const char *refs_check_dir, +struct dir_iterator *iter); + +static int files_fsck_refs_dir (struct ref_store *ref_store, +struct fsck_options *o, +const char *refs_check_dir, +files_fsck_refs_fn *fsck_refs_fns) +{ +const char *gitdir = ref_store->gitdir; +struct strbuf sb =STRBUF_INIT ; +struct dir_iterator *iter; +int iter_status; +int ret =0 ; + +strbuf_addf (&sb,"%s/%s" , gitdir, refs_check_dir); + + iter =dir_iterator_begin (sb.buf,0 ); + +if (!iter) { + ret =error_errno ("cannot open directory %s" , sb.buf); + goto out; + } + +while ((iter_status =dir_iterator_advance (iter)) ==ITER_OK ) { +if (S_ISDIR (iter->st.st_mode)) { +continue ; + }else if (S_ISREG (iter->st.st_mode) || +S_ISLNK (iter->st.st_mode)) { +if (o->verbose) +fprintf_ln (stderr,"Checking %s/%s" , + refs_check_dir, iter->relative_path); +for (size_t i =0 ; fsck_refs_fns[i]; i++) { +if (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter)) + ret =-1 ; + } + }else { + ret =error (_ ("unexpected file type for '%s'" ), + iter->basename); + } + } + +if (iter_status !=ITER_DONE ) + ret =error (_ ("failed to iterate over '%s'" ), sb.buf); + +out: +strbuf_release (&sb); +return ret; +} + +static int files_fsck_refs (struct ref_store *ref_store, +struct fsck_options *o) +{ +files_fsck_refs_fn fsck_refs_fns[]= { +NULL + }; + +if (o->verbose) +fprintf_ln (stderr,"Checking references consistency" ); + +return files_fsck_refs_dir (ref_store, o,"refs" , fsck_refs_fns); + +} +static int files_fsck (struct ref_store *ref_store,struct fsck_options *o) {struct files_ref_store *refs =files_downcast (ref_store,REF_STORE_READ ,"fsck" ); -return refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); +return refs->packed_ref_store->be->fsck (refs->packed_ref_store, o) | +files_fsck_refs (ref_store, o); }struct ref_storage_be refs_be_files = {
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v13 08/10] files-backend: add unified interface for refs scanning
Quoted 47 lines · shejialuo
For refs and reflogs, we need to scan its corresponding directories to check every regular file or symbolic link which shares the same pattern. Introduce a unified interface for scanning directories for files-backend.
Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- refs/files-backend.c | 74 +++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 73 insertions(+), 1 deletion(-)refs/files-backend.c +20
-0@@ -6,6 +6,7 @@#include "../gettext.h" #include "../hash.h" #include "../hex.h" +#include "../fsck.h" #include "../refs.h" #include "refs-internal.h" #include "ref-cache.h" @@ -3408,13 +3409,84 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +/* +* For refs and reflogs, they share a unified interface when scanning +* the whole directory. This function is used as the callback for each +* regular file or symlink in the directory. +*/ +typedef int (*files_fsck_refs_fn )(struct fsck_options *o, +const char *gitdir, +const char *refs_check_dir, +struct dir_iterator *iter); + +static int files_fsck_refs_dir (struct ref_store *ref_store, +struct fsck_options *o, +const char *refs_check_dir, +files_fsck_refs_fn *fsck_refs_fns) +{ +const char *gitdir = ref_store->gitdir; +struct strbuf sb =STRBUF_INIT ; +struct dir_iterator *iter; +int iter_status;
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3440
Quoted 10 lines
refs/files-backend.c +10
-0+int ret =0 ; + +strbuf_addf (&sb,"%s/%s" , gitdir, refs_check_dir); + + iter =dir_iterator_begin (sb.buf,0 ); + +if (!iter) { + ret =error_errno ("cannot open directory %s" , sb.buf); + goto out; + }
The error message should probably be marked as translatable. Also, I'd personally remove the newline between iter = ... and the error check as those are a logical unit.
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3449
Quoted 8 lines
refs/files-backend.c +8
-0+while ((iter_status =dir_iterator_advance (iter)) ==ITER_OK ) { +if (S_ISDIR (iter->st.st_mode)) { +continue ; + }else if (S_ISREG (iter->st.st_mode) || +S_ISLNK (iter->st.st_mode)) { +if (o->verbose) +fprintf_ln (stderr,"Checking %s/%s" , + refs_check_dir, iter->relative_path);
Okay, we do end up using the verbose flag :)
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3456
Quoted 7 lines
refs/files-backend.c +7
-0+for (size_t i =0 ; fsck_refs_fns[i]; i++) { +if (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter)) + ret =-1 ; + } + }else { + ret =error (_ ("unexpected file type for '%s'" ), + iter->basename);
Instead of printing this as an error directly, shouldn't we report it via the fsck_refs_report interface?
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3457
Quoted 2 lines
refs/files-backend.c +2
-0+ } + }
Okay. It does make sense to do our own directory walk as that will allow us to check files which would otherwise not be reported by the normal refs interfaces.
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3461
Quoted 2 lines
refs/files-backend.c +2
-0+if (iter_status !=ITER_DONE ) + ret =error (_ ("failed to iterate over '%s'" ), sb.buf);
Reporting this as an error feels sensible though as we have no ref to tie this error to, and it feels like a generic error.
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3472
Quoted 10 lines
refs/files-backend.c +10
-0+out: +strbuf_release (&sb); +return ret; +} + +static int files_fsck_refs (struct ref_store *ref_store, +struct fsck_options *o) +{ +files_fsck_refs_fn fsck_refs_fns[]= { +NULL
The last member should also end with a comma.
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3479
Quoted 7 lines
refs/files-backend.c +7
-0+ }; + +if (o->verbose) +fprintf_ln (stderr,"Checking references consistency" ); + +return files_fsck_refs_dir (ref_store, o,"refs" , fsck_refs_fns); +
This newline should be removed.
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3489
Quoted 11 lines
refs/files-backend.c +4
-1+} +static int files_fsck (struct ref_store *ref_store,struct fsck_options *o) {struct files_ref_store *refs =files_downcast (ref_store,REF_STORE_READ ,"fsck" ); -return refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); +return refs->packed_ref_store->be->fsck (refs->packed_ref_store, o) | +files_fsck_refs (ref_store, o);
I'd think we should first check loose files and then continue to check the packed ref store. That's really only a gut feeling though, and I cannot exactly say why that feels more natural.
Patrick
Re: [GSoC][PATCH v13 08/10] files-backend: add unified interface for refs scanning
Quoted 11 lines
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3440
Quoted 6 lines
refs/files-backend.c +6
-0+ iter =dir_iterator_begin (sb.buf,0 ); + +if (!iter) { + ret =error_errno ("cannot open directory %s" , sb.buf); + goto out; + }The error message should probably be marked as translatable. Also, I'd personally remove the newline between
iter = ...and the error check as those are a logical unit.
Yes, I will improve this in the next version.
Quoted 11 lines
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3456
Quoted 7 lines
refs/files-backend.c +7
-0+for (size_t i =0 ; fsck_refs_fns[i]; i++) { +if (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter)) + ret =-1 ; + } + }else { + ret =error (_ ("unexpected file type for '%s'" ), + iter->basename);Instead of printing this as an error directly, shouldn't we report it via the
fsck_refs_reportinterface?
Yes, exactly we should use this interface. I accidentally ignored this. Thanks.
Quoted 4 lines
Quoted 4 lines
refs/files-backend.c +4
-0+out: +strbuf_release (&sb); +return ret; +}
Quoted 9 lines
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3472
Quoted 6 lines
refs/files-backend.c +6
-0+ +static int files_fsck_refs (struct ref_store *ref_store, +struct fsck_options *o) +{ +files_fsck_refs_fn fsck_refs_fns[]= { +NULL The last member should also end with a comma.
I will improve this in the next version.
Quoted 10 lines
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3479
Quoted 7 lines
refs/files-backend.c +7
-0+ }; + +if (o->verbose) +fprintf_ln (stderr,"Checking references consistency" ); + +return files_fsck_refs_dir (ref_store, o,"refs" , fsck_refs_fns); +This newline should be removed.
OK.
Quoted 8 lines
Quoted 8 lines
refs/files-backend.c +2
-0+} +static int files_fsck (struct ref_store *ref_store,struct fsck_options *o) {struct files_ref_store *refs =files_downcast (ref_store,REF_STORE_READ ,"fsck" );
Quoted 8 lines
refs/files-backend.c static int files_ref_store_remove_on_disk(struct ref_store *ref_store, · line 3489
Quoted 3 lines
refs/files-backend.c +2
-1-return refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); +return refs->packed_ref_store->be->fsck (refs->packed_ref_store, o) | +files_fsck_refs (ref_store, o);I'd think we should first check loose files and then continue to check the packed ref store. That's really only a gut feeling though, and I cannot exactly say why that feels more natural.
It would feel more natural. Because packed-ref will point to the loose ref. So we should first check the loose refs. For example. If a regular ref is bad, we will first report the problem. And suppose the packed-ref have recorded this regular ref. When checking packed-ref, we could not check the regular ref itself. We only need to check one thing.
Whether the pointee exists under the "refs/" directory.
And we do not need to check regular ref again because we have checked in the loose refs part.
Quoted 1 line
Patrick
PATCH v13 9/10 fsck: add ref name check for files backend
In reply to shejialuoThe git-fsck(1) only implicitly checks the reference, it does not fully check refs with bad format name such as standalone "@" and name ending with ".lock".
In order to provide such checks, add a new fsck message id "badRefName" with default ERROR type. Use existing "check_refname_format" to explicit check the ref name. And add a new unit test to verify the functionality.
--- Documentation/fsck-msgids.txt | 3 + fsck.h | 1 + refs/files-backend.c | 22 ++++++++ t/t0602-reffiles-fsck.sh | 101 ++++++++++++++++++++++++++++++++++ 4 files changed, 127 insertions(+) create mode 100755 t/t0602-reffiles-fsck.sh
Documentation/fsck-msgids.txt +3 -0
@@ -19,6 +19,9 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefName`:: + (ERROR) A ref has an invalid format. + `badTagName`:: (INFO) A tag has an invalid format.
fsck.h +1 -0
@@ -31,6 +31,7 @@ enum fsck_msg_type { FUNC(BAD_NAME, ERROR) \ FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ + FUNC(BAD_REF_NAME, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
refs/files-backend.c +22 -0
@@ -3419,6 +3419,27 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,const char *refs_check_dir, struct dir_iterator *iter); +static int files_fsck_refs_name (struct fsck_options *o, + const char *gitdir UNUSED, +const char *refs_check_dir, +struct dir_iterator *iter) +{ +struct strbuf sb =STRBUF_INIT ; +struct fsck_refs_info info; +int ret =0 ; + +if (check_refname_format (iter->basename,REFNAME_ALLOW_ONELEVEL )) { +strbuf_addf (&sb,"%s/%s" , refs_check_dir, iter->relative_path); + info.path = sb.buf; + ret =fsck_refs_report (o,NULL , &info, +FSCK_MSG_BAD_REF_NAME , +"invalid refname format" ); + } + +strbuf_release (&sb); +return ret; +} + static int files_fsck_refs_dir(struct ref_store *ref_store, struct fsck_options *o, const char *refs_check_dir,
@@ -3469,6 +3490,7 @@ static int files_fsck_refs(struct ref_store *ref_store, struct fsck_options *o) {files_fsck_refs_fn fsck_refs_fns[]= { + files_fsck_refs_name,NULL };
t/t0602-reffiles-fsck.sh new file +101 -0
101 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v13 09/10] fsck: add ref name check for files backend
Quoted 33 lines · shejialuo
refs/files-backend.c +21
-0@@ -3419,6 +3419,27 @@ typedef int (*files_fsck_refs_fn)(struct fsck_options *o,const char *refs_check_dir, struct dir_iterator *iter); +static int files_fsck_refs_name (struct fsck_options *o, + const char *gitdir UNUSED, +const char *refs_check_dir, +struct dir_iterator *iter) +{ +struct strbuf sb =STRBUF_INIT ; +struct fsck_refs_info info; +int ret =0 ; + +if (check_refname_format (iter->basename,REFNAME_ALLOW_ONELEVEL )) { +strbuf_addf (&sb,"%s/%s" , refs_check_dir, iter->relative_path); + info.path = sb.buf; + ret =fsck_refs_report (o,NULL , &info, +FSCK_MSG_BAD_REF_NAME , +"invalid refname format" ); + } + +strbuf_release (&sb); +return ret; +} + static int files_fsck_refs_dir(struct ref_store *ref_store, struct fsck_options *o, const char *refs_check_dir,
refs/files-backend.c static int files_fsck_refs(struct ref_store *ref_store, · line 3494
Quoted 5 lines
refs/files-backend.c +1
-0@@ -3469,6 +3490,7 @@ static int files_fsck_refs(struct ref_store *ref_store, struct fsck_options *o) {files_fsck_refs_fn fsck_refs_fns[]= { + files_fsck_refs_name,NULL
Neat. I very much like that we can simply add new checks to this function and the rest is handled for us already. Makes this whole thing nicely extensible.
Quoted 8 lines
refs/files-backend.c +0
-0};t/t0602-reffiles-fsck.sh new file +0
-0
t/t0602-reffiles-fsck.sh line 10
Quoted 10 lines
t/t0602-reffiles-fsck.sh new file +10
-0@@ -0,0 +1,101 @@ +#!/bin/sh + +test_description='Test reffiles backend consistency check' + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME +GIT_TEST_DEFAULT_REF_FORMAT=files +export GIT_TEST_DEFAULT_REF_FORMAT + +. ./test-lib.sh
Is this test suite intentionally not marked withTEST_PASSES_SANITIZE_LEAK=true?
Quoted 6 lines
t/t0602-reffiles-fsck.sh new file +6
-0+ +test_expect_success 'ref name should be checked' ' + test_when_finished "rm -rf repo" && + git init repo && + branch_dir_prefix=.git/refs/heads && + tag_dir_prefix=.git/refs/tags &&
t/t0602-reffiles-fsck.sh line 26
Quoted 10 lines
t/t0602-reffiles-fsck.sh new file +10
-0+ ( + cd repo && + git commit --allow-empty -m initial && + git checkout -b branch-1 && + git tag tag-1 && + git commit --allow-empty -m second && + git checkout -b branch-2 && + git tag tag-2 && + git tag multi_hierarchy/tag-2 + ) &&
I don't quite get why you create several subshells only to cd into repo in each of them. Isn't a single subshell sufficient for all of those tests? If you want to delimit blocks, then you can simply add an empty newline between them.
Quoted 10 lines
t/t0602-reffiles-fsck.sh new file +10
-0+ ( + cd repo && + cp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 && + test_must_fail git fsck 2>err && + cat >expect <<-EOF && + error: refs/heads/.branch-1: badRefName: invalid refname format + EOF + rm $branch_dir_prefix/.branch-1 && + test_cmp expect err + ) &&
t/t0602-reffiles-fsck.sh line 46
Quoted 10 lines
t/t0602-reffiles-fsck.sh new file +10
-0+ ( + cd repo && + cp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock && + test_must_fail git fsck 2>err && + cat >expect <<-EOF && + error: refs/tags/tag-1.lock: badRefName: invalid refname format + EOF + rm $tag_dir_prefix/tag-1.lock && + test_cmp expect err + ) &&
The other cases all make sense, but I don't think that a file ending with ".lock" should be marked as having a "badRefName". It is expected that concurrent writers may have such lock files.
What could make sense is to eventually mark stale lock files older than X amount of time as errors or warnings. But I'd think that this is outside of the scope of this patch series.
Patrick
Re: [GSoC][PATCH v13 09/10] fsck: add ref name check for files backend
Quoted 11 lines
Quoted 11 lines
t/t0602-reffiles-fsck.sh new file +5
-0@@ -0,0 +1,101 @@ +#!/bin/sh + +test_description='Test reffiles backend consistency check' + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
Quoted 9 lines
t/t0602-reffiles-fsck.sh line 10
Quoted 5 lines
t/t0602-reffiles-fsck.sh new file +5
-0+export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME +GIT_TEST_DEFAULT_REF_FORMAT=files +export GIT_TEST_DEFAULT_REF_FORMAT + +. ./test-lib.shIs this test suite intentionally not marked with
TEST_PASSES_SANITIZE_LEAK=true?
No, I don't know this. I will add TEST_PASSES_SANITIZE_LEAK=true and export this environment variable.
Quoted 13 lines
Quoted 13 lines
t/t0602-reffiles-fsck.sh new file +13
-0+ +test_expect_success 'ref name should be checked' ' + test_when_finished "rm -rf repo" && + git init repo && + branch_dir_prefix=.git/refs/heads && + tag_dir_prefix=.git/refs/tags && + ( + cd repo && + git commit --allow-empty -m initial && + git checkout -b branch-1 && + git tag tag-1 && + git commit --allow-empty -m second && + git checkout -b branch-2 &&
Quoted 9 lines
t/t0602-reffiles-fsck.sh line 26
Quoted 3 lines
t/t0602-reffiles-fsck.sh new file +3
-0+ git tag tag-2 && + git tag multi_hierarchy/tag-2 + ) &&I don't quite get why you create several subshells only to cd into
repoin each of them. Isn't a single subshell sufficient for all of those tests? If you want to delimit blocks, then you can simply add an empty newline between them.
I just want to delimit, I will use newline in the next version.
Quoted 25 lines
Quoted 10 lines
t/t0602-reffiles-fsck.sh new file +10
-0+ ( + cd repo && + cp $branch_dir_prefix/branch-1 $branch_dir_prefix/.branch-1 && + test_must_fail git fsck 2>err && + cat >expect <<-EOF && + error: refs/heads/.branch-1: badRefName: invalid refname format + EOF + rm $branch_dir_prefix/.branch-1 && + test_cmp expect err + ) &&t/t0602-reffiles-fsck.sh line 46
Quoted 10 lines
t/t0602-reffiles-fsck.sh new file +10
-0+ ( + cd repo && + cp $tag_dir_prefix/tag-1 $tag_dir_prefix/tag-1.lock && + test_must_fail git fsck 2>err && + cat >expect <<-EOF && + error: refs/tags/tag-1.lock: badRefName: invalid refname format + EOF + rm $tag_dir_prefix/tag-1.lock && + test_cmp expect err + ) &&The other cases all make sense, but I don't think that a file ending with ".lock" should be marked as having a "badRefName". It is expected that concurrent writers may have such lock files.
Quoted 4 lines
What could make sense is to eventually mark stale lock files older than X amount of time as errors or warnings. But I'd think that this is outside of the scope of this patch series.
If so, let us just ignore ".lock" situation at the moment.
Quoted 1 line
Patrick
PATCH v13 10/10 fsck: add ref content check for files backend
In reply to shejialuoEnhance the git-fsck(1) command by adding a check for reference content in the files backend. The new functionality ensures that symrefs, real symbolic link and regular refs are validated correctly.
In order to check the trailing content of the regular refs, add a new parameter trailing to parse_loose_ref_contents.
For symrefs, parse_loose_ref_contents will set the "referent". However, symbolic link could be either absolute or relative. Use "strbuf_add_real_path" to read the symbolic link and convert the relative path to absolute path. Then use "skip_prefix" to make it align with symref "referent".
Thus, the symrefs and symbolic links could share the same interface. Add a new function "files_fsck_symref_target" which aims at checking the following things:
1. whether the pointee is under the refs/ directory.
2. whether the pointee name is correct.
3. whether the pointee path is a wrong type in filesystem.
Last, add the following FSCK MESSAGEs:
1. "badRefContent(ERROR)": A ref has a bad content
2. "badSymrefPointee(ERROR)": The pointee of a symref is bad.
3. "trailingRefContent(WARN)": A ref content has trailing contents.
--- Documentation/fsck-msgids.txt | 9 +++ fsck.h | 3 + refs.c | 2 +- refs/files-backend.c | 144 +++++++++++++++++++++++++++++++++- refs/refs-internal.h | 5 +- t/t0602-reffiles-fsck.sh | 110 ++++++++++++++++++++++++++ 6 files changed, 268 insertions(+), 5 deletions(-)
Documentation/fsck-msgids.txt +9 -0
@@ -19,9 +19,15 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefContent`:: + (ERROR) A ref has a bad content. + `badRefName`:: (ERROR) A ref has an invalid format. +`badSymrefPointee`:: + (ERROR) The pointee of a symref is bad. + `badTagName`:: (INFO) A tag has an invalid format.
@@ -167,6 +173,9 @@ `nullSha1`:: (WARN) Tree contains entries pointing to a null sha1. +`trailingRefContent`:: + (WARN) A ref content has trailing contents. + `treeNotSorted`:: (ERROR) A tree is not properly sorted.
fsck.h +3 -0
@@ -32,6 +32,8 @@ enum fsck_msg_type { FUNC(BAD_OBJECT_SHA1, ERROR) \ FUNC(BAD_PARENT_SHA1, ERROR) \ FUNC(BAD_REF_NAME, ERROR) \ + FUNC(BAD_REF_CONTENT, ERROR) \ + FUNC(BAD_SYMREF_POINTEE, ERROR) \ FUNC(BAD_TIMEZONE, ERROR) \ FUNC(BAD_TREE, ERROR) \FUNC (BAD_TREE_SHA1 ,ERROR ) \
@@ -72,6 +74,7 @@ enum fsck_msg_type { FUNC(HAS_DOTDOT, WARN) \ FUNC(HAS_DOTGIT, WARN) \ FUNC(NULL_SHA1, WARN) \ + FUNC(TRAILING_REF_CONTENT, WARN) \ FUNC(ZERO_PADDED_FILEMODE, WARN) \ FUNC(NUL_IN_COMMIT, WARN) \FUNC (LARGE_PATHNAME ,WARN ) \
refs.c +1 -1
@@ -1760,7 +1760,7 @@ static int refs_read_special_head(struct ref_store *ref_store, } result =parse_loose_ref_contents (content.buf, oid, referent, type, - failure_errno); + failure_errno,NULL ); done:strbuf_release (&full_path);
refs/files-backend.c +142 -2
180 lines not on this page · raw message
refs/refs-internal.h +3 -2
@@ -709,11 +709,12 @@ struct ref_store {/* * Parse contents of a loose ref file. *failure_errno maybe be set to EINVAL for -* invalid contents. +* invalid contents. Also *trailing is set to the first character after the +* refname or NULL if the referent is not empty. */ int parse_loose_ref_contents (const char *buf,struct object_id *oid,struct strbuf *referent,unsigned int *type, -int *failure_errno); +int *failure_errno,const char **trailing);/* * Fill in the generic part of refs and add it to our collection of
t/t0602-reffiles-fsck.sh +110 -0
114 lines not on this page · raw message
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v13 10/10] fsck: add ref content check for files backend
Quoted 21 lines · shejialuo
Enhance the git-fsck(1) command by adding a check for reference content in the files backend. The new functionality ensures that symrefs, real symbolic link and regular refs are validated correctly.
In order to check the trailing content of the regular refs, add a new parameter
trailingtoparse_loose_ref_contents.For symrefs,
parse_loose_ref_contentswill set the "referent". However, symbolic link could be either absolute or relative. Use "strbuf_add_real_path" to read the symbolic link and convert the relative path to absolute path. Then use "skip_prefix" to make it align with symref "referent".Thus, the symrefs and symbolic links could share the same interface. Add a new function "
files_fsck_symref_target" which aims at checking the following things:1. whether the pointee is under the
refs/directory.
2. whether the pointee name is correct.
3. whether the pointee path is a wrong type in filesystem.
Quoted 5 lines
Last, add the following FSCK MESSAGEs:
1. "badRefContent(ERROR)": A ref has a bad content
2. "badSymrefPointee(ERROR)": The pointee of a symref is bad.
3. "trailingRefContent(WARN)": A ref content has trailing contents.
I think it would have been fine to stop at the preceding commit as it clearly demonstrates how the whole infrastructure is supposed to work. Additional checks like those you add here would then be a good candidate for a separate patch series. This would help you get the first patch series landed faster as you really only have to focus on setting up the baseline infrastructure.
Feel free to keep or drop this patch as you prefer though, I don't want to discourage you aiming higher. Just keep in mind that the more you add on top the longer it takes to land a patch series :)
Documentation/fsck-msgids.txt line 24
Quoted 11 lines
Documentation/fsck-msgids.txt +3
-0@@ -19,9 +19,15 @@ `badParentSha1`:: (ERROR) A commit object has a bad parent sha1. +`badRefContent`:: + (ERROR) A ref has a bad content. +
s/a bad/bad
Documentation/fsck-msgids.txt
Quoted 10 lines
Documentation/fsck-msgids.txt +10
-0+static int files_fsck_refs_content(struct fsck_options *o, + const char *gitdir, + const char *refs_check_dir, + struct dir_iterator *iter) +{ + struct strbuf pointee_path = STRBUF_INIT, + ref_content = STRBUF_INIT, + abs_gitdir = STRBUF_INIT, + referent = STRBUF_INIT, + refname = STRBUF_INIT;
Nit: I think it's more customary to start each of the lines with struct strbuf. Not a 100% certain on this one, though.
Quoted 55 lines
Documentation/fsck-msgids.txt +55
-0+ const char *trailing = NULL; + struct fsck_refs_info info; + int failure_errno = 0; + unsigned int type = 0; + struct object_id oid; + int ret = 0; + + strbuf_addf(&refname, "%s/%s", refs_check_dir, iter->relative_path); + info.path = refname.buf; + + /* + * If the file is a symlink, we need to only check the connectivity + * of the destination object. + */ + if (S_ISLNK(iter->st.st_mode)) { + const char *pointee_name = NULL; + + strbuf_add_real_path(&pointee_path, iter->path.buf); + + strbuf_add_absolute_path(&abs_gitdir, gitdir); + strbuf_normalize_path(&abs_gitdir); + if (!is_dir_sep(abs_gitdir.buf[abs_gitdir.len - 1])) + strbuf_addch(&abs_gitdir, '/'); + + if (!skip_prefix(pointee_path.buf, + abs_gitdir.buf, &pointee_name)) { + ret = fsck_refs_report(o, NULL, &info, + FSCK_MSG_BAD_SYMREF_POINTEE, + "point to target outside gitdir"); + goto clean; + } + + ret = files_fsck_symref_target(o, &info, refname.buf, + pointee_name, pointee_path.buf); + goto clean; + } + + if (strbuf_read_file(&ref_content, iter->path.buf, 0) < 0) { + ret = error_errno(_("%s/%s: unable to read the ref"), + refs_check_dir, iter->relative_path); + goto clean; + } + + if (parse_loose_ref_contents(ref_content.buf, &oid, + &referent, &type, + &failure_errno, &trailing)) { + ret = fsck_refs_report(o, NULL, &info, + FSCK_MSG_BAD_REF_CONTENT, + "invalid ref content"); + goto clean; + } + + /* + * If the ref is a symref, we need to check the destination name and + * connectivity.
Documentation/fsck-msgids.txt
Quoted 10 lines
Documentation/fsck-msgids.txt +10
-0+ */ + if (referent.len && (type & REF_ISSYMREF)) { + strbuf_addf(&pointee_path, "%s/%s", gitdir, referent.buf); + strbuf_rtrim(&referent); + + ret = files_fsck_symref_target(o, &info, refname.buf, + referent.buf, pointee_path.buf); + goto clean; + } else { + if (trailing && (*trailing != '\0' && *trailing != '\n')) {
In case the ref ends with a newline, should we check that the next character is \0? Otherwise, it may contain multiple lines, which is not allowed for a normal ref.
Also, shouldn't the ref always end with a newline?
Patrick
Re: [GSoC][PATCH v13 10/10] fsck: add ref content check for files backend
Quoted 35 lines · Patrick Steinhardt
Quoted 21 lines · shejialuo
Enhance the git-fsck(1) command by adding a check for reference content in the files backend. The new functionality ensures that symrefs, real symbolic link and regular refs are validated correctly.
In order to check the trailing content of the regular refs, add a new parameter
trailingtoparse_loose_ref_contents.For symrefs,
parse_loose_ref_contentswill set the "referent". However, symbolic link could be either absolute or relative. Use "strbuf_add_real_path" to read the symbolic link and convert the relative path to absolute path. Then use "skip_prefix" to make it align with symref "referent".Thus, the symrefs and symbolic links could share the same interface. Add a new function "
files_fsck_symref_target" which aims at checking the following things:1. whether the pointee is under the
refs/directory.
2. whether the pointee name is correct.
3. whether the pointee path is a wrong type in filesystem.Quoted 5 lines
Last, add the following FSCK MESSAGEs:
1. "badRefContent(ERROR)": A ref has a bad content
2. "badSymrefPointee(ERROR)": The pointee of a symref is bad.
3. "trailingRefContent(WARN)": A ref content has trailing contents.I think it would have been fine to stop at the preceding commit as it clearly demonstrates how the whole infrastructure is supposed to work. Additional checks like those you add here would then be a good candidate for a separate patch series. This would help you get the first patch series landed faster as you really only have to focus on setting up the baseline infrastructure.
Quoted 4 lines
Feel free to keep or drop this patch as you prefer though, I don't want to discourage you aiming higher. Just keep in mind that the more you add on top the longer it takes to land a patch series :)
I will drop this patch in the next version. Actually, in the very former version, I didn't realise that the effort to set up the infra is so much.
Quoted 18 lines
refs/files-backend.c static int files_fsck_refs_name(struct fsck_options *o, · line 3565
Quoted 13 lines
refs/files-backend.c +13
-0+/* +* If the ref is a symref, we need to check the destination name and +* connectivity. +*/ +if (referent.len && (type &REF_ISSYMREF )) { +strbuf_addf (&pointee_path,"%s/%s" , gitdir, referent.buf); +strbuf_rtrim (&referent); + + ret =files_fsck_symref_target (o, &info, refname.buf, + referent.buf, pointee_path.buf); + goto clean; + }else { +if (trailing && (*trailing !='\0' && *trailing !='\n' )) {In case the ref ends with a newline, should we check that the next character is
\0? Otherwise, it may contain multiple lines, which is not allowed for a normal ref.
Quoted 2 lines
Also, shouldn't the ref always end with a newline?
This is a very interesting question here. Based on my experiments, I have found the following things:
It's OK that regular refs contain multiple newlines. And git totally allows such case. The current code does not handle multiple newlines.
For symrefs, it allows spaces and newlines, for example:
ref: refs/heads/master <space>
ref: refs/heads/master \n\n\n
But for such case, git will report error:
ref: refs/heads/master garbage
And ref can be end with a newline or not. Both will be accepted. Junio have told me that there is no spec really. So, I ignore multiple newlines for regular refs and also ignore multiple newlines and trailing spaces for symref.
Quoted 1 line
Patrick
Re: [GSoC][PATCH v13 10/10] fsck: add ref content check for files backend
In reply to shejialuorefs/files-backend.c static int files_fsck_refs_name(struct fsck_options *o, · line 3546
Quoted 9 lines · shejialuo
refs/files-backend.c +9
-0+static int files_fsck_refs_content (struct fsck_options *o, +const char *gitdir, +const char *refs_check_dir, +struct dir_iterator *iter) +{ + ... +if (parse_loose_ref_contents (ref_content.buf, &oid, + &referent, &type, + &failure_errno, &trailing)) {
The function parse_loose_ref_contents() needs to know what the hash algorithm is, and it used to implicitly assume that the_repository's hash algorithm was an OK thing to use. Patrick's recent clean-up series instead passes "struct ref_store *refs" throughout the call chain so that "ref->repo->hash_algo" can be used. This needs some matching change, which means ...
refs/files-backend.c static int files_fsck_refs(struct ref_store *ref_store, · line 3635
Quoted 5 lines
refs/files-backend.c +1
-0files_fsck_refs_fn fsck_refs_fns[]= { files_fsck_refs_name, + files_fsck_refs_content,NULL };
... the function signature for files_fsck_refs_fn must change to have something that lets you access repo->hash_algo.
By the way, unless the most common use of an array is to pass it around as a collection of items and operate on the collection, it is a better practice to name an array with a singular noun. Name the array as fsck_refs_fn[] not _fns[]. This is so that you can refer to a single element in a more grammatical way. E.g. with
struct dog dog[] = { { .breed="shiba" }, { .breed="beagle" } };you can say "dog[0] has brown fur" instead of "dogs[0] has ...".
In this case, you do not treat the collection of functions as a one thing and do something to the collection. Instead you'd repeat over the functions in a loop and individually call them, perhaps like so:
for (i = 0; fsck_fn[i] != NULL; i++) fsck_fn[i](...);
so it is very much more appropriate to name the array itself as singular to allow you to say "first fsck_fn", "next fsck_fn", etc.
Re: [GSoC][PATCH v13 10/10] fsck: add ref content check for files backend
Quoted 19 lines · Junio C Hamano
refs/files-backend.c static int files_fsck_refs_name(struct fsck_options *o, · line 3546
Quoted 9 lines · shejialuo
refs/files-backend.c +9
-0+static int files_fsck_refs_content (struct fsck_options *o, +const char *gitdir, +const char *refs_check_dir, +struct dir_iterator *iter) +{ + ... +if (parse_loose_ref_contents (ref_content.buf, &oid, + &referent, &type, + &failure_errno, &trailing)) {The function
parse_loose_ref_contents()needs to know what the hash algorithm is, and it used to implicitly assume that the_repository's hash algorithm was an OK thing to use. Patrick's recent clean-up series instead passes "structref_store*refs" throughout the call chain so that "ref->repo->hash_algo" can be used. This needs some matching change, which means ...
Quoted 9 lines
refs/files-backend.c static int files_fsck_refs(struct ref_store *ref_store, · line 3635
Quoted 5 lines
refs/files-backend.c +1
-0files_fsck_refs_fn fsck_refs_fns[]= { files_fsck_refs_name, + files_fsck_refs_content,NULL };... the function signature for
files_fsck_refs_fnmust change to have something that lets you accessrepo->hash_algo.
Thanks for your remind, I have scanned this patch:
https://lore.kernel.org/git/fe0e2c3617c8040c632dbc3de613a1d22e8070f7.1722316795.git.ps@pks.im/
I guess I will handle this later. It seems that this series has not come into the cooking tree. I will update this part until Patrick's patch gets merged into "next".
Quoted 15 lines
By the way, unless the most common use of an array is to pass it around as a collection of items and operate on the collection, it is a better practice to name an array with a singular noun. Name the array as fsck_refs_fn[] not _fns[]. This is so that you can refer to a single element in a more grammatical way. E.g. with
struct dog dog[] = { { .breed="shiba" }, { .breed="beagle" } };you can say "dog[0] has brown fur" instead of "dogs[0] has ...".
In this case, you do not treat the collection of functions as a one thing and do something to the collection. Instead you'd repeat over the functions in a loop and individually call them, perhaps like so:
Quoted 6 lines
for (i = 0; fsck_fn[i] != NULL; i++) fsck_fn[i](...);so it is very much more appropriate to name the array itself as singular to allow you to say "first
fsck_fn", "nextfsck_fn", etc.
Thanks, I have learned a lot here. I will improve this in the next version.
Quoted 28 lines · shejialuo
Hi All:
This version mainly focuses on the problem about how should we provide extensibility for reporting the problem of refs. In patch v10, Junio asked a question here:
Quoted 11 lines
The error reporting function for refs consistency check was still about reporting a problem for a single ref. I am wondering how consistency violations that are not about a single ref should be handled. For example, if refs/packed-backend.c:packed_fsck() finds that the file is not sorted properly or has some unparseable garbage in it, it is not something you can report as "refs/heads/main is broken", but those who are interested in seeing the "reference database consistency" verified, it is very much what they want the tool to notice. How would detection of such a breakage that is not attributed to a single ref fit in this "ref consistency check infrastructure" that was introduced by [05/10]?
Actually, I think that the original parameter "
checked_ref_name" is a bad name which makes the reader think that "we only handle refs". And this is my fault for this design. However, I misunderstood the Junio's word and made things complicated.The patch v11 and v12 wants to solve a problem that we should provide extensibility for reporting refs problem. However, these two versions still made things complicated. After an offline meeting with Patrick and Karthik, we design the following simple flat data structure:
Quoted 5 lines
struct fsck_refs_info { const char *path; };It is simple and provides extensibility.
I've got a bunch of comments, mostly because I've been out for quite a while and thus didn't follow the progression of this series. But don't let yourself be discouraged by the volume, I quite like the shape of this patch series and think that it is very close to what we actually want to have.
I'd say that this series is likely to become ready soonish.
Thanks!
Patrick
Quoted 11 lines
I've got a bunch of comments, mostly because I've been out for quite a while and thus didn't follow the progression of this series. But don't let yourself be discouraged by the volume, I quite like the shape of this patch series and think that it is very close to what we actually want to have.
I'd say that this series is likely to become ready soonish.
Thanks!
Patrick
That's OK, Patrick. It's important for me to receive reviews thus we can make this series better. And I will wait for the advice from the Junio about the design of the "fsck_error" here. If Junio were satisfied with this design. I will clean the code and submit a new version until everything is OK.
Thanks,
Jialuo
v14 cover ref consistency check infra setup
In reply to shejialuoHi All,
This version does the following things:
1. By following the advice from Patrick, we should make the callback function be generic by adding only one "void * fsck_report" parameter. Thus the commit sequence will be much more clearer. And it wll be much easier for reviewers to review. And I have split the commit into more commits in this version.
2. Enhance the commit messages to provide more context about why we should do this.
3. Patrick advices that we should initialize the "fsck_options" member when parsing the options. However, because the original "strict" and "verbose" field are defined as the bit field, we cannot take the address of them. So I simply remove the bit field.
4. As Patrick said, ".lock" should not be reported as error. At current, ignore files ending with ".lock".
5. Add a fsck msg type called "badRefFiletype" which indicates that a ref has a bad file type when scanning the directory.
6. Junio advices instead of using "fsck_refs_fns", we should use the singular version "fsck_refs_fn", fix this.
7. Drop the last patch because in this series, we mainly focus on the infra, I will add a series later to add ref content check.
However, there is one thing holding. Junio advices that I should follow the Patrick's change to change the prototype of "files_fsck_refs_fn"
https://lore.kernel.org/git/fe0e2c3617c8040c632dbc3de613a1d22e8070f7.1722316795.git.ps@pks.im/
However, at current, this patch is not in the cooking tree. I will handle this later.
shejialuo (11):
fsck: rename "skiplist" to "skip_oids"
fsck: make "fsck_error" callback generic
fsck: add a unified interface for reporting fsck messages
fsck: add refs report function
fsck: add refs-related error callback
fsck: rename objects-related fsck error functions
refs: set up ref consistency check infrastructure builtin/refs: add verify subcommand builtin/fsck: add git-refs verify child process
files-backend: add unified interface for refs scanning
fsck: add ref name check for files backend
Documentation/fsck-msgids.txt | 6 ++ Documentation/git-refs.txt | 13 ++++ builtin/fsck.c | 34 +++++++-- builtin/mktag.c | 3 +- builtin/refs.c | 34 +++++++++ fsck.c | 131 +++++++++++++++++++++++++++------- fsck.h | 76 +++++++++++++++----- object-file.c | 9 ++- refs.c | 5 ++ refs.h | 8 +++ refs/debug.c | 11 +++ refs/files-backend.c | 118 +++++++++++++++++++++++++++++- refs/packed-backend.c | 8 +++ refs/refs-internal.h | 6 ++ refs/reftable-backend.c | 8 +++ t/t0602-reffiles-fsck.sh | 94 ++++++++++++++++++++++++ 16 files changed, 504 insertions(+), 60 deletions(-) create mode 100755 t/t0602-reffiles-fsck.sh
Range-diff against v13:
1: 772cad5b92 = 1: 75b64a219d fsck: rename "skiplist" to "skip_oids" 2: e76449977d < -: ---------- fsck: add a unified interface for reporting fsck messages -: ---------- > 2: 81433d1628 fsck: make "fsck_error" callback generic -: ---------- > 3: 0792c51e6d fsck: add a unified interface for reporting fsck messages -: ---------- > 4: ecd144af15 fsck: add refs report function -: ---------- > 5: e80dba0cab fsck: add refs-related error callback 3: 26bde4283f ! 6: a61db42bf2 fsck: rename objects-related fsck error functions
@@ Metadata
## Commit message ##
fsck: rename objects-related fsck error functions
- The names of objects-related fsck error functions are general. It's OK
+ The names of objects-related fsck error functions are generic. It's OK
when there is only object database check. However, we have introduced refs database check report function. To avoid ambiguity, rename object-related fsck error functions to explicitly indicate these
@@ builtin/fsck.c: static int objerror(struct object *obj, const char *err)
}
-static int fsck_error_func(struct fsck_options *o UNUSED,
-- const struct object_id *oid,
-- enum object_type object_type,
-- const struct fsck_refs_info *refs_info UNUSED,
+- void *fsck_report,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+static int fsck_objects_error_func(struct fsck_options *o UNUSED,
-+ const struct object_id *oid,
-+ enum object_type object_type,
-+ const struct fsck_refs_info *refs_info UNUSED,
++ void *fsck_report,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
{
- switch (msg_type) {
- case FSCK_WARN:
+ struct fsck_object_report *report = fsck_report;
+ const struct object_id *oid = report->oid;
@@ builtin/fsck.c: int cmd_fsck(int argc, const char **argv, const char *prefix)
fsck_walk_options.walk = mark_object;
@@ fsck.c: int fsck_buffer(const struct object_id *oid, enum object_type type,
}
-int fsck_error_function(struct fsck_options *o,
-- const struct object_id *oid,
-- enum object_type object_type UNUSED,
-- const struct fsck_refs_info *refs_info UNUSED,
+- void *fsck_report,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id UNUSED,
- const char *message)
+int fsck_objects_error_function(struct fsck_options *o,
-+ const struct object_id *oid,
-+ enum object_type object_type UNUSED,
-+ const struct fsck_refs_info *refs_info UNUSED,
++ void *fsck_report,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id UNUSED,
+ const char *message)
{
- if (msg_type == FSCK_WARN) {
- warning("object %s: %s", fsck_describe_object(o, oid), message);
+ struct fsck_object_report *report = fsck_report;
+ const struct object_id *oid = report->oid;
@@ fsck.c: int git_fsck_config(const char *var, const char *value,
* Custom error callbacks that are used in more than one place. */
-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
-- const struct object_id *oid,
-- enum object_type object_type,
-- const struct fsck_refs_info *refs_info,
+- void *fsck_report,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message)
+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
-+ const struct object_id *oid,
-+ enum object_type object_type,
-+ const struct fsck_refs_info *refs_info,
++ void *fsck_report,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message)
{
if (msg_id == FSCK_MSG_GITMODULES_MISSING) {
- puts(oid_to_hex(oid));
+ struct fsck_object_report *report = fsck_report;
+ puts(oid_to_hex(report->oid));
return 0;
}
-- return fsck_error_function(o, oid, object_type, refs_info,
-- msg_type, msg_id, message);
-+ return fsck_objects_error_function(o, oid, object_type, refs_info,
-+ msg_type, msg_id, message);
+- return fsck_error_function(o, fsck_report, msg_type, msg_id, message);
++ return fsck_objects_error_function(o, fsck_report, msg_type,msg_id, message);
}
## fsck.h ##
@@ fsck.h: typedef int (*fsck_error)(struct fsck_options *o,
const char *message);
-int fsck_error_function(struct fsck_options *o,
-- const struct object_id *oid, enum object_type object_type,
-- const struct fsck_refs_info *refs_info,
+- void *fsck_report,
- enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
- const char *message);
-int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,
-- const struct object_id *oid,
-- enum object_type object_type,
-- const struct fsck_refs_info *refs_info,
+- void *fsck_report,
- enum fsck_msg_type msg_type,
- enum fsck_msg_id msg_id,
- const char *message);
+int fsck_objects_error_function(struct fsck_options *o,
-+ const struct object_id *oid, enum object_type object_type,
-+ const struct fsck_refs_info *refs_info,
++ void *fsck_report,
+ enum fsck_msg_type msg_type, enum fsck_msg_id msg_id,
+ const char *message);
+int fsck_objects_error_cb_print_missing_gitmodules(struct fsck_options *o,
-+ const struct object_id *oid,
-+ enum object_type object_type,
-+ const struct fsck_refs_info *refs_info,
++ void *fsck_report,
+ enum fsck_msg_type msg_type,
+ enum fsck_msg_id msg_id,
+ const char *message);
- /*
- * The information for reporting refs-related error message
+ int fsck_refs_error_function(struct fsck_options *options,
+ void *fsck_report,
@@ fsck.h: struct fsck_options {
.gitmodules_done = OIDSET_INIT, \
.gitattributes_found = OIDSET_INIT, \
@@ fsck.h: struct fsck_options {
- .error_func = fsck_error_cb_print_missing_gitmodules, \
+ .error_func = fsck_objects_error_cb_print_missing_gitmodules, \
}
-
- /* descend in all linked child objects
+ #define FSCK_REFS_OPTIONS_DEFAULT { \
+ .error_func = fsck_refs_error_function, \
4: 56fc833a54 < -: ---------- fsck: add refs-related error report function 5: 90f992ecfb ! 7: f2248bc52d refs: set up ref consistency check infrastructure
@@ Metadata
## Commit message ##
refs: set up ref consistency check infrastructure
+ The "struct ref_store" is the base class which contains the "be" pointer
+ which provides backend-specific functions whose interfaces are defined
+ in the "ref_storage_be". We could reuse this polymorphism to define only
+ one interface. For every backend, we need to provide its own function
+ pointer.
+
The interfaces defined in the ref_storage_be are carefully structured in semantic. It's organized as the five parts:
6: b2277f5ef4 ! 8: 0c5463d757 git refs: add verify subcommand
@@ Metadata
Author: shejialuo <[email protected]>
## Commit message ##
- git refs: add verify subcommand
+ builtin/refs: add verify subcommand
Introduce a new subcommand "verify" in git-refs(1) to allow the user to
check the reference database consistency and also this subcommand will
- be used as the entry point of checking refs for "git-fsck(1)". Last, add
- "verbose" field into "fsck_options" to indicate whether we should print
- verbose messages when checking refs and objects consistency.
+ be used as the entry point of checking refs for "git-fsck(1)".
+
+ Add "verbose" field into "fsck_options" to indicate whether we should
+ print verbose messages when checking refs and objects consistency.
+
+ Remove bit-field for "strict" field, this is because we cannot take
+ address of a bit-field which makes it unhandy to set member variables
+ when parsing the command line options.
+
+ The "git-fsck(1)" declares "fsck_options" variable with "static"
+ identifier which avoids complaint by the leak-checker. However, in
+ "git-refs verify", we need to do memory clean manually. Thus add
+ "fsck_options_clear" function in "fsck.c" to provide memory clean
+ operation.
Mentored-by: Patrick Steinhardt <[email protected]>
Mentored-by: Karthik Nayak <[email protected]>
@@ Documentation/git-refs.txt: include::ref-storage-format.txt[]
+The following options are specific to 'git refs verify':
+
+--strict::
-+ Enable more strict checking, every WARN severity for the `Fsck Messages`
-+ be seen as ERROR. See linkgit:git-fsck[1].
++ Enable stricter error checking. This will cause warnings to be
++ reported as errors. See linkgit:git-fsck[1].
+
+--verbose::
+ When verifying the reference database consistency, be chatty.
@@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c
+ REFS_VERIFY_USAGE,
+ NULL,
+ };
-+ unsigned int verbose = 0, strict = 0;
+ struct option options[] = {
-+ OPT__VERBOSE(&verbose, N_("be verbose")),
-+ OPT_BOOL(0, "strict", &strict, N_("enable strict checking")),
++ OPT_BOOL(0, "verbose", &fsck_refs_options.verbose, N_("be verbose")),
++ OPT_BOOL(0, "strict", &fsck_refs_options.strict, N_("enable strict checking")),
+ OPT_END(),
+ };
+ int ret;
@@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c
+ if (argc)
+ usage(_("'git refs verify' takes no arguments"));
+
-+ if (verbose)
-+ fsck_refs_options.verbose = 1;
-+ if (strict)
-+ fsck_refs_options.strict = 1;
-+
+ git_config(git_fsck_config, &fsck_refs_options);
+ prepare_repo_settings(the_repository);
+
+ ret = refs_fsck(get_main_ref_store(the_repository), &fsck_refs_options);
+
-+ /*
-+ * Explicitly free the allocated array and "skip_oids" set
-+ */
-+ free(fsck_refs_options.msg_type);
-+ oidset_clear(&fsck_refs_options.skip_oids);
++ fsck_options_clear(&fsck_refs_options);
+ return ret;
+}
+
@@ builtin/refs.c: static int cmd_refs_migrate(int argc, const char **argv, const c
};
+ ## fsck.c ##
+@@ fsck.c: int fsck_finish(struct fsck_options *options)
+ return ret;
+ }
+
++void fsck_options_clear(struct fsck_options *options)
++{
++ free(options->msg_type);
++ oidset_clear(&options->skip_oids);
++ oidset_clear(&options->gitmodules_found);
++ oidset_clear(&options->gitmodules_done);
++ oidset_clear(&options->gitattributes_found);
++ oidset_clear(&options->gitattributes_done);
++ kh_clear_oid_map(options->object_names);
++}
++
+ int git_fsck_config(const char *var, const char *value,
+ const struct config_context *ctx, void *cb)
+ {
+
## fsck.h ##
-@@ fsck.h: struct fsck_options {
+@@ fsck.h: struct fsck_ref_report {
+ struct fsck_options {
fsck_walk_func walk;
fsck_error error_func;
- unsigned strict:1;
-+ unsigned verbose:1;
+- unsigned strict:1;
++ unsigned strict;
++ unsigned verbose;
enum fsck_msg_type *msg_type;
struct oidset skip_oids;
struct oidset gitmodules_found;
+@@ fsck.h: int fsck_tag_standalone(const struct object_id *oid, const char *buffer,
+ */
+ int fsck_finish(struct fsck_options *options);
+
++/*
++ * Clear the fsck_options struct, freeing any allocated memory.
++ */
++void fsck_options_clear(struct fsck_options *options);
++
+ /*
+ * Report an error or warning for refs.
+ */
7: f96d0f200d = 9: 0ac25c2f6e builtin/fsck: add `git-refs verify` child process
8: f9a0c16bef ! 10: 2b6f5e6c46 files-backend: add unified interface for refs scanning
@@ Commit message
Mentored-by: Karthik Nayak <[email protected]>
Signed-off-by: shejialuo <[email protected]>
+ ## Documentation/fsck-msgids.txt ##
+@@
+ `badParentSha1`::
+ (ERROR) A commit object has a bad parent sha1.
+
++`badRefFiletype`::
++ (ERROR) A ref has a bad file type.
++
+ `badTagName`::
+ (INFO) A tag has an invalid format.
+
+
+ ## fsck.h ##
+@@ fsck.h: enum fsck_msg_type {
+ FUNC(BAD_NAME, ERROR) \
+ FUNC(BAD_OBJECT_SHA1, ERROR) \
+ FUNC(BAD_PARENT_SHA1, ERROR) \
++ FUNC(BAD_REF_FILETYPE, ERROR) \
+ FUNC(BAD_TIMEZONE, ERROR) \
+ FUNC(BAD_TREE, ERROR) \
+ FUNC(BAD_TREE_SHA1, ERROR) \
+
## refs/files-backend.c ##
@@
#include "../gettext.h"
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
+static int files_fsck_refs_dir(struct ref_store *ref_store,
+ struct fsck_options *o,
+ const char *refs_check_dir,
-+ files_fsck_refs_fn *fsck_refs_fns)
++ files_fsck_refs_fn *fsck_refs_fn)
+{
+ const char *gitdir = ref_store->gitdir;
+ struct strbuf sb = STRBUF_INIT;
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
+ strbuf_addf(&sb, "%s/%s", gitdir, refs_check_dir);
+
+ iter = dir_iterator_begin(sb.buf, 0);
-+
+ if (!iter) {
-+ ret = error_errno("cannot open directory %s", sb.buf);
++ ret = error_errno(_("cannot open directory %s"), sb.buf);
+ goto out;
+ }
+
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
+ if (o->verbose)
+ fprintf_ln(stderr, "Checking %s/%s",
+ refs_check_dir, iter->relative_path);
-+ for (size_t i = 0; fsck_refs_fns[i]; i++) {
-+ if (fsck_refs_fns[i](o, gitdir, refs_check_dir, iter))
++ for (size_t i = 0; fsck_refs_fn[i]; i++) {
++ if (fsck_refs_fn[i](o, gitdir, refs_check_dir, iter))
+ ret = -1;
+ }
+ } else {
-+ ret = error(_("unexpected file type for '%s'"),
-+ iter->basename);
++ struct fsck_ref_report report = { .path = iter->basename };
++ if (fsck_report_ref(o, &report,
++ FSCK_MSG_BAD_REF_FILETYPE,
++ "unexpected file type"))
++ ret = -1;
+ }
+ }
+
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
+static int files_fsck_refs(struct ref_store *ref_store,
+ struct fsck_options *o)
+{
-+ files_fsck_refs_fn fsck_refs_fns[]= {
-+ NULL
++ files_fsck_refs_fn fsck_refs_fn[]= {
++ NULL,
+ };
+
+ if (o->verbose)
+ fprintf_ln(stderr, "Checking references consistency");
-+
-+ return files_fsck_refs_dir(ref_store, o, "refs", fsck_refs_fns);
++ return files_fsck_refs_dir(ref_store, o, "refs", fsck_refs_fn);
+
+}
+
@@ refs/files-backend.c: static int files_ref_store_remove_on_disk(struct ref_store
files_downcast(ref_store, REF_STORE_READ, "fsck");
- return refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);
-+ return refs->packed_ref_store->be->fsck(refs->packed_ref_store, o) |
-+ files_fsck_refs(ref_store, o);
++ return files_fsck_refs(ref_store, o) |
++ refs->packed_ref_store->be->fsck(refs->packed_ref_store, o);
}
struct ref_storage_be refs_be_files = {
9: ee55ee5787 < -: ---------- fsck: add ref name check for files backend
10: 9256328cbb < -: ---------- fsck: add ref content check for files backend
-: ---------- > 11: 14f2739bd7 fsck: add ref name check for files backendSignature · 1 line
2.45.2
PATCH v14 1/11 fsck: rename "skiplist" to "skip_oids"
The "skiplist" field in "fsck_options" is related to objects. Because we are going to introduce ref consistency check, the "skiplist" name is too general which will make the caller think "skiplist" is related to both the refs and objects.
It may seem that for both refs and objects, we should provide a general "skiplist" here. However, the type for "skiplist" is struct oidset which is totally unsuitable for refs.
To avoid above ambiguity, rename "skiplist" to "skip_oids".
--- fsck.c | 4 ++-- fsck.h | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
fsck.c +2 -2
@@ -205,7 +205,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)if (!strcmp (buf,"skiplist" )) {if (equal == len)die ("skiplist requires a path" ); -oidset_parse_file (&options->skiplist, buf + equal +1 , +oidset_parse_file (&options->skip_oids, buf + equal +1 , the_repository->hash_algo); buf += len +1 ;continue ;
@@ -223,7 +223,7 @@ void fsck_set_msg_types(struct fsck_options *options, const char *values)static int object_on_skiplist (struct fsck_options *opts,const struct object_id *oid) { -return opts && oid &&oidset_contains (&opts->skiplist, oid); +return opts && oid &&oidset_contains (&opts->skip_oids, oid); } __attribute__((format (printf, 5, 6)))
fsck.h +2 -2
@@ -136,7 +136,7 @@ struct fsck_options {fsck_error error_func; unsigned strict:1;enum fsck_msg_type *msg_type; -struct oidset skiplist; +struct oidset skip_oids;struct oidset gitmodules_found;struct oidset gitmodules_done;struct oidset gitattributes_found;
@@ -145,7 +145,7 @@ struct fsck_options { };#define FSCK_OPTIONS_DEFAULT { \ - .skiplist = OIDSET_INIT, \ + .skip_oids = OIDSET_INIT, \ .gitmodules_found = OIDSET_INIT, \ .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \
Signature · 1 line
2.45.2
PATCH v14 2/11 fsck: make "fsck_error" callback generic
In reply to shejialuoThe "fsck_error" callback is designed to report the objects-related error messages. It accepts two parameter "oid" and "object_type" which is not generic. In order to provide a unified callback which can report either objects or refs, remove the objects-related parameters and add the generic parameter "void *fsck_report".
Create a new "fsck_object_report" structure which incorporates the removed parameters "oid" and "object_type". Then change the corresponding references to adapt to new "fsck_error" callback.
--- builtin/fsck.c | 7 +++++-- builtin/mktag.c | 3 +-- fsck.c | 20 +++++++++++++------- fsck.h | 17 ++++++++++++----- object-file.c | 9 ++++----- 5 files changed, 35 insertions(+), 21 deletions(-)
builtin/fsck.c +5 -2
@@ -90,12 +90,15 @@ static int objerror(struct object *obj, const char *err) } static int fsck_error_func(struct fsck_options *o UNUSED, -const struct object_id *oid, -enum object_type object_type, +void *fsck_report,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message) { +struct fsck_object_report *report = fsck_report; +const struct object_id *oid = report->oid; +enum object_type object_type = report->object_type; +switch (msg_type) {case FSCK_WARN :/* TRANSLATORS: e.g. warning in tree 01bfda: <more explanation> */
builtin/mktag.c +1 -2
@@ -18,8 +18,7 @@ static int option_strict = 1;static struct fsck_options fsck_options =FSCK_OPTIONS_STRICT ; static int mktag_fsck_error_func(struct fsck_options *o UNUSED, - const struct object_id *oid UNUSED, - enum object_type object_type UNUSED, + void *fsck_report UNUSED,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message)
fsck.c +13 -7
@@ -232,6 +232,10 @@ static int report(struct fsck_options *options, enum fsck_msg_id msg_id, const char *fmt, ...) { va_list ap; +struct fsck_object_report report = { + .oid = oid, + .object_type = object_type + };struct strbuf sb =STRBUF_INIT ;enum fsck_msg_type msg_type =fsck_msg_type (msg_id, options);int result;
@@ -252,7 +256,7 @@ static int report(struct fsck_options *options,va_start (ap, fmt);strbuf_vaddf (&sb, fmt, ap); - result = options->error_func (options, oid, object_type, + result = options->error_func (options, &report, msg_type, msg_id, sb.buf);strbuf_release (&sb);va_end (ap);
@@ -1201,12 +1205,14 @@ int fsck_buffer(const struct object_id *oid, enum object_type type, }int fsck_error_function (struct fsck_options *o, -const struct object_id *oid, - enum object_type object_type UNUSED, +void *fsck_report,enum fsck_msg_type msg_type, enum fsck_msg_id msg_id UNUSED,const char *message) { +struct fsck_object_report *report = fsck_report; +const struct object_id *oid = report->oid; +if (msg_type ==FSCK_WARN ) {warning ("object %s: %s" ,fsck_describe_object (o, oid), message);return 0 ;
@@ -1304,15 +1310,15 @@ int git_fsck_config(const char *var, const char *value,*/ int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, +void *fsck_report,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message) {if (msg_id ==FSCK_MSG_GITMODULES_MISSING ) { -puts (oid_to_hex (oid)); +struct fsck_object_report *report = fsck_report; +puts (oid_to_hex (report->oid));return 0 ; } -return fsck_error_function (o, oid, object_type, msg_type, msg_id, message); +return fsck_error_function (o, fsck_report, msg_type, msg_id, message); }
fsck.h +12 -5
@@ -114,23 +114,30 @@ int is_valid_msg_type(const char *msg_id, const char *msg_type);typedef int (*fsck_walk_func )(struct object *obj,enum object_type object_type,void *data,struct fsck_options *options); -/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */ +/* +* Callback for reporting errors either for objects or refs. The "fsck_report" +* is a generic pointer that can be used to pass any information. +*/ typedef int (*fsck_error )(struct fsck_options *o, -const struct object_id *oid,enum object_type object_type, +void *fsck_report,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);int fsck_error_function (struct fsck_options *o, -const struct object_id *oid,enum object_type object_type, +void *fsck_report,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message);int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -const struct object_id *oid, -enum object_type object_type, +void *fsck_report,enum fsck_msg_type msg_type,enum fsck_msg_id msg_id,const char *message); +struct fsck_object_report { +const struct object_id *oid; +enum object_type object_type; +}; +struct fsck_options {fsck_walk_func walk;fsck_error error_func;
object-file.c +4 -5
@@ -2470,11 +2470,10 @@ int repo_has_object_file(struct repository *r,* give more context. */ static int hash_format_check_report(struct fsck_options *opts UNUSED, - const struct object_id *oid UNUSED, - enum object_type object_type UNUSED, - enum fsck_msg_type msg_type UNUSED, - enum fsck_msg_id msg_id UNUSED, -const char *message) + void *fsck_report UNUSED, + enum fsck_msg_type msg_type UNUSED, + enum fsck_msg_id msg_id UNUSED, +const char *message) {error (_ ("object fails fsck: %s" ), message);return 1 ;
Signature · 1 line
2.45.2
PATCH v14 3/11 fsck: add a unified interface for reporting fsck messages
In reply to shejialuoThe static function "report" provided by "fsck.c" aims at checking error type and calling the callback "error_func" to report the message. Both refs and objects need to check the error type of the current fsck message. In order to extract this common behavior, create a new function "fsck_vreport". Instead of using "...", provide "va_list" to allow more flexibility.
Instead of changing "report" prototype to be algin with the "fsck_vreport" function, we leave the "report" prototype unchanged due to the reason that there are nearly 62 references about "report" function. Simply change "report" function to use "fsck_vreport" to report objects related messages.
--- fsck.c | 49 ++++++++++++++++++++++++++++++++++--------------- 1 file changed, 34 insertions(+), 15 deletions(-)
fsck.c +34 -15
@@ -226,16 +226,16 @@ static int object_on_skiplist(struct fsck_options *opts,return opts && oid &&oidset_contains (&opts->skip_oids, oid); } -__attribute__((format (printf,5 ,6 ))) -static int report (struct fsck_options *options, -const struct object_id *oid,enum object_type object_type, -enum fsck_msg_id msg_id,const char *fmt, ...) +/* +* Provide the common functionality for either fscking refs or objects. +* It will get the current msg error type and call the error_func callback +* which is registered in the "fsck_options" struct. +*/ +static int fsck_vreport (struct fsck_options *options, +void *fsck_report, +enum fsck_msg_id msg_id,const char *fmt,va_list ap) { -va_list ap; -struct fsck_object_report report = { - .oid = oid, - .object_type = object_type - }; +va_list ap_copy;struct strbuf sb =STRBUF_INIT ;enum fsck_msg_type msg_type =fsck_msg_type (msg_id, options);int result;
@@ -243,9 +243,6 @@ static int report(struct fsck_options *options,if (msg_type ==FSCK_IGNORE )return 0 ; -if (object_on_skiplist (options, oid)) -return 0 ; -if (msg_type ==FSCK_FATAL ) msg_type =FSCK_ERROR ;else if (msg_type ==FSCK_INFO )
@@ -254,9 +251,9 @@ static int report(struct fsck_options *options,prepare_msg_ids ();strbuf_addf (&sb,"%s: " , msg_id_info[msg_id].camelcased); -va_start (ap, fmt); -strbuf_vaddf (&sb, fmt, ap); - result = options->error_func (options, &report, +va_copy (ap_copy, ap); +strbuf_vaddf (&sb, fmt, ap_copy); + result = options->error_func (options, fsck_report, msg_type, msg_id, sb.buf);strbuf_release (&sb);va_end (ap);
@@ -264,6 +261,28 @@ static int report(struct fsck_options *options,return result; } +__attribute__((format (printf,5 ,6 ))) +static int report (struct fsck_options *options, +const struct object_id *oid,enum object_type object_type, +enum fsck_msg_id msg_id,const char *fmt, ...) +{ +va_list ap; +struct fsck_object_report report = { + .oid = oid, + .object_type = object_type + }; +int result; + +if (object_on_skiplist (options, oid)) +return 0 ; + +va_start (ap, fmt); + result =fsck_vreport (options, &report, msg_id, fmt, ap); +va_end (ap); + +return result; +} +void fsck_enable_object_names (struct fsck_options *options) {if (!options->object_names)
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v14 03/11] fsck: add a unified interface for reporting fsck messages
fsck.c static int report(struct fsck_options *options, · line 254
Quoted 8 lines · shejialuo
fsck.c +1
-3@@ -254,9 +251,9 @@ static int report(struct fsck_options *options,prepare_msg_ids ();strbuf_addf (&sb,"%s: " , msg_id_info[msg_id].camelcased); -va_start (ap, fmt); -strbuf_vaddf (&sb, fmt, ap); - result = options->error_func(options, &report, +va_copy (ap_copy, ap);
Can't we use ap directly instead of copying it? We'd have to get rid of the call to va_end as our caller already does that, but other than that I don't see any reason to copy the argument list here.
fsck.c static int report(struct fsck_options *options, · line 259
Quoted 5 lines
fsck.c +2
-0+strbuf_vaddf (&sb, fmt, ap_copy); + result = options->error_func (options, fsck_report, msg_type, msg_id, sb.buf);strbuf_release (&sb);va_end (ap);
Patrick
Re: [GSoC][PATCH v14 03/11] fsck: add a unified interface for reporting fsck messages
Quoted 5 lines · Patrick Steinhardt
Quoted 4 lines · shejialuo
fsck.c +0
-0@@ -254,9 +251,9 @@ static int report(struct fsck_options *options,prepare_msg_ids ();strbuf_addf (&sb,"%s: " , msg_id_info[msg_id].camelcased);
Quoted 9 lines
fsck.c static int report(struct fsck_options *options, · line 254
Quoted 4 lines
fsck.c +1
-3-va_start (ap, fmt); -strbuf_vaddf (&sb, fmt, ap); - result = options->error_func(options, &report, +va_copy (ap_copy, ap);Can't we use
apdirectly instead of copying it? We'd have to get rid of the call tova_endas our caller already does that, but other than that I don't see any reason to copy the argument list here.
Thanks, Patrick! This is right. I will fix this in the next version.
Quoted 7 lines
fsck.c static int report(struct fsck_options *options, · line 259
Quoted 5 lines
fsck.c +2
-0+strbuf_vaddf (&sb, fmt, ap_copy); + result = options->error_func (options, fsck_report, msg_type, msg_id, sb.buf);strbuf_release (&sb);va_end (ap);Patrick
PATCH v14 4/11 fsck: add refs report function
In reply to shejialuoIntroduce a new struct "fsck_ref_report" to contain the information we need when reporting refs-related messages.
With the new "fsck_vreport" function, add a new function "fsck_report_ref" to report refs-related fsck error message. Unlike "report" function uses the exact parameters, we simply pass "struct fsck_ref_report *report" as the parameter. This is because at current we don't know exactly how many fields we need. By passing this parameter, we don't need to change this function prototype when we want to add more information into "fsck_ref_report".
--- fsck.c | 13 +++++++++++++ fsck.h | 16 ++++++++++++++++ 2 files changed, 29 insertions(+)
fsck.c +13 -0
@@ -283,6 +283,19 @@ static int report(struct fsck_options *options,return result; } +int fsck_report_ref (struct fsck_options *options, +struct fsck_ref_report *report, +enum fsck_msg_id msg_id, +const char *fmt, ...) +{ +va_list ap; +int result; +va_start (ap, fmt); + result =fsck_vreport (options, report, msg_id, fmt, ap); +va_end (ap); +return result; +} +void fsck_enable_object_names (struct fsck_options *options) {if (!options->object_names)
fsck.h +16 -0
@@ -138,6 +138,12 @@ struct fsck_object_report {enum object_type object_type; }; +struct fsck_ref_report { +const char *path; +const struct object_id *oid; +const char *referent; +}; +struct fsck_options {fsck_walk_func walk;fsck_error error_func;
@@ -216,6 +222,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +/* +* Report an error or warning for refs. +*/ +__attribute__((format (printf,4 ,5 ))) +int fsck_report_ref (struct fsck_options *options, +struct fsck_ref_report *report, +enum fsck_msg_id msg_id, +const char *fmt, ...); + +/* * Subsystem for storing human-readable names for each object. *
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v14 04/11] fsck: add refs report function
fsck.h int fsck_tag_standalone(const struct object_id *oid, const char *buffer, · line 232
Quoted 12 lines · shejialuo
fsck.h +8
-0@@ -216,6 +222,16 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +/* +* Report an error or warning for refs. +*/ +__attribute__((format (printf,4 ,5 ))) +int fsck_report_ref (struct fsck_options *options, +struct fsck_ref_report *report, +enum fsck_msg_id msg_id, +const char *fmt, ...);
Nice that we got rid of the parts that don't matter for reporting refs now.
Patrick
PATCH v14 5/11 fsck: add refs-related error callback
In reply to shejialuoWe have introduced "fsck_report_ref" function to report the error message for refs. We still need to add the corresponding callback function. Create refs-specific "error_func" callback "fsck_refs_error_function".
Last, add "FSCK_REFS_OPTIONS_DEFAULT" macro to create default options when checking ref consistency.
--- fsck.c | 28 ++++++++++++++++++++++++++++ fsck.h | 9 +++++++++ 2 files changed, 37 insertions(+)
fsck.c +28 -0
@@ -1253,6 +1253,34 @@ int fsck_error_function(struct fsck_options *o,return 1 ; } +int fsck_refs_error_function(struct fsck_options *options UNUSED, +void *fsck_report, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) +{ +struct fsck_ref_report *report = fsck_report; + +struct strbuf sb =STRBUF_INIT ; +int ret =0 ; + +strbuf_addstr (&sb, report->path); + +if (report->oid) +strbuf_addf (&sb," -> (%s)" ,oid_to_hex (report->oid)); +else if (report->referent) +strbuf_addf (&sb," -> (%s)" , report->referent); + +if (msg_type ==FSCK_WARN ) +warning ("%s: %s" , sb.buf, message); +else + ret =error ("%s: %s" , sb.buf, message); + +strbuf_release (&sb); +return ret; + +} +static int fsck_blobs (struct oidset *blobs_found,struct oidset *blobs_done,enum fsck_msg_id msg_missing,enum fsck_msg_id msg_type,struct fsck_options *options,const char *blob_type)
fsck.h +9 -0
@@ -133,6 +133,12 @@ int fsck_error_cb_print_missing_gitmodules(struct fsck_options *o,enum fsck_msg_id msg_id, const char *message); +int fsck_refs_error_function (struct fsck_options *options, +void *fsck_report, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message); +struct fsck_object_report {const struct object_id *oid;enum object_type object_type;
@@ -181,6 +187,9 @@ struct fsck_options { .gitattributes_done = OIDSET_INIT, \ .error_func = fsck_error_cb_print_missing_gitmodules, \ } +#define FSCK_REFS_OPTIONS_DEFAULT { \ + .error_func = fsck_refs_error_function, \ +}/* descend in all linked child objects * the return value is:
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v14 05/11] fsck: add refs-related error callback
Quoted 7 lines · shejialuo
We have introduced "
fsck_report_ref" function to report the error message for refs. We still need to add the corresponding callback function. Create refs-specific "error_func" callback "fsck_refs_error_function".Last, add "
FSCK_REFS_OPTIONS_DEFAULT" macro to create default options when checking ref consistency.
Nit: personally, I'd squash this commit into the preceding one. They belong together logically, as the preceding step doesn't make any sense without the error reporting function and this step here doesn't make any sense without introducing struct fsck_ref_report.
Patrick
PATCH v14 6/11 fsck: rename objects-related fsck error functions
In reply to shejialuoThe names of objects-related fsck error functions are generic. It's OK when there is only object database check. However, we have introduced refs database check report function. To avoid ambiguity, rename object-related fsck error functions to explicitly indicate these functions are used to report objects-related messages.
--- builtin/fsck.c | 12 ++++++------ fsck.c | 22 +++++++++++----------- fsck.h | 24 ++++++++++++------------ 3 files changed, 29 insertions(+), 29 deletions(-)
builtin/fsck.c +6 -6
@@ -89,11 +89,11 @@ static int objerror(struct object *obj, const char *err)return -1 ; } -static int fsck_error_func(struct fsck_options *o UNUSED, -void *fsck_report, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +static int fsck_objects_error_func(struct fsck_options *o UNUSED, +void *fsck_report, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {struct fsck_object_report *report = fsck_report;const struct object_id *oid = report->oid;
@@ -941,7 +941,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) fsck_walk_options.walk = mark_object; fsck_obj_options.walk = mark_used; - fsck_obj_options.error_func = fsck_error_func; + fsck_obj_options.error_func = fsck_objects_error_func;if (check_strict) fsck_obj_options.strict =1 ;
fsck.c +11 -11
@@ -1236,11 +1236,11 @@ int fsck_buffer(const struct object_id *oid, enum object_type type, type); } -int fsck_error_function (struct fsck_options *o, -void *fsck_report, -enum fsck_msg_type msg_type, - enum fsck_msg_id msg_id UNUSED, -const char *message) +int fsck_objects_error_function (struct fsck_options *o, +void *fsck_report, +enum fsck_msg_type msg_type, + enum fsck_msg_id msg_id UNUSED, +const char *message) {struct fsck_object_report *report = fsck_report;const struct object_id *oid = report->oid;
@@ -1369,16 +1369,16 @@ int git_fsck_config(const char *var, const char *value,* Custom error callbacks that are used in more than one place. */ -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -void *fsck_report, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message) +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +void *fsck_report, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message) {if (msg_id ==FSCK_MSG_GITMODULES_MISSING ) {struct fsck_object_report *report = fsck_report;puts (oid_to_hex (report->oid));return 0 ; } -return fsck_error_function (o, fsck_report, msg_type, msg_id, message); +return fsck_objects_error_function (o, fsck_report, msg_type,msg_id, message); }
fsck.h +12 -12
@@ -123,15 +123,15 @@ typedef int (*fsck_error)(struct fsck_options *o, enum fsck_msg_type msg_type, enum fsck_msg_id msg_id, const char *message); -int fsck_error_function (struct fsck_options *o, -void *fsck_report, -enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, -const char *message); -int fsck_error_cb_print_missing_gitmodules (struct fsck_options *o, -void *fsck_report, -enum fsck_msg_type msg_type, -enum fsck_msg_id msg_id, -const char *message); +int fsck_objects_error_function (struct fsck_options *o, +void *fsck_report, +enum fsck_msg_type msg_type,enum fsck_msg_id msg_id, +const char *message); +int fsck_objects_error_cb_print_missing_gitmodules (struct fsck_options *o, +void *fsck_report, +enum fsck_msg_type msg_type, +enum fsck_msg_id msg_id, +const char *message); int fsck_refs_error_function(struct fsck_options *options, void *fsck_report,
@@ -169,7 +169,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function \ + .error_func = fsck_objects_error_function \ }#define FSCK_OPTIONS_STRICT { \ .strict = 1, \
@@ -177,7 +177,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_function, \ + .error_func = fsck_objects_error_function, \ }#define FSCK_OPTIONS_MISSING_GITMODULES { \ .strict = 1, \
@@ -185,7 +185,7 @@ struct fsck_options { .gitmodules_done = OIDSET_INIT, \ .gitattributes_found = OIDSET_INIT, \ .gitattributes_done = OIDSET_INIT, \ - .error_func = fsck_error_cb_print_missing_gitmodules, \ + .error_func = fsck_objects_error_cb_print_missing_gitmodules, \ }#define FSCK_REFS_OPTIONS_DEFAULT { \ .error_func = fsck_refs_error_function, \
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v14 06/11] fsck: rename objects-related fsck error functions
Quoted 5 lines · shejialuo
The names of objects-related fsck error functions are generic. It's OK when there is only object database check. However, we have introduced refs database check report function. To avoid ambiguity, rename object-related fsck error functions to explicitly indicate these functions are used to report objects-related messages.
Nit: I think it would be a bit easier to follow if you moved this step a bit earlier, namely before you introduce the ref-related infra.
Patrick
PATCH v14 7/11 refs: set up ref consistency check infrastructure
In reply to shejialuoThe "struct ref_store" is the base class which contains the "be" pointer which provides backend-specific functions whose interfaces are defined in the "ref_storage_be". We could reuse this polymorphism to define only one interface. For every backend, we need to provide its own function pointer.
The interfaces defined in the ref_storage_be are carefully structured in semantic. It's organized as the five parts:
1. The name and the initialization interfaces.
2. The ref transaction interfaces.
3. The ref internal interfaces (pack, rename and copy).
4. The ref filesystem interfaces.
5. The reflog related interfaces.
To keep consistent with the git-fsck(1), add a new interface named "fsck_refs_fn" to the end of "ref_storage_be". This semantic cannot be grouped into any above five categories. Explicitly add blank line to make it different from others.
Last, implement placeholder functions for each ref backends.
--- refs.c | 5 +++++ refs.h | 8 ++++++++ refs/debug.c | 11 +++++++++++ refs/files-backend.c | 13 ++++++++++++- refs/packed-backend.c | 8 ++++++++ refs/refs-internal.h | 6 ++++++ refs/reftable-backend.c | 8 ++++++++ 7 files changed, 58 insertions(+), 1 deletion(-)
refs.c +5 -0
@@ -318,6 +318,11 @@ int check_refname_format(const char *refname, int flags)return check_or_sanitize_refname (refname, flags,NULL ); } +int refs_fsck (struct ref_store *refs,struct fsck_options *o) +{ +return refs->be->fsck (refs, o); +} +void sanitize_refname_component (const char *refname,struct strbuf *out) {if (check_or_sanitize_refname (refname,REFNAME_ALLOW_ONELEVEL , out))
refs.h +8 -0
@@ -4,6 +4,7 @@#include "commit.h" #include "repository.h" +struct fsck_options ;struct object_id ;struct ref_store ;struct strbuf ;
@@ -541,6 +542,13 @@ int refs_for_each_reflog(struct ref_store *refs, each_reflog_fn fn, void *cb_dat*/ int check_refname_format (const char *refname,int flags); +/* +* Check the reference database for consistency. Return 0 if refs and +* reflogs are consistent, and non-zero otherwise. The errors will be +* written to stderr. +*/ +int refs_fsck (struct ref_store *refs,struct fsck_options *o); +/* * Apply the rules from check_refname_format, but mutate the result until it * is acceptable, and place the result in "out".
refs/debug.c +11 -0
@@ -419,6 +419,15 @@ static int debug_reflog_expire(struct ref_store *ref_store, const char *refname,return res; } +static int debug_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +struct debug_ref_store *drefs = (struct debug_ref_store *)ref_store; +int res = drefs->refs->be->fsck (drefs->refs, o); +trace_printf_key (&trace_refs,"fsck: %d\n" , res); +return res; +} +struct ref_storage_be refs_be_debug = { .name ="debug" , .init =NULL ,
@@ -451,4 +460,6 @@ struct ref_storage_be refs_be_debug = { .create_reflog = debug_create_reflog, .delete_reflog = debug_delete_reflog, .reflog_expire = debug_reflog_expire, + + .fsck = debug_fsck, };
refs/files-backend.c +12 -1
@@ -3408,6 +3408,15 @@ static int files_ref_store_remove_on_disk(struct ref_store *ref_store,return ret; } +static int files_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +struct files_ref_store *refs = +files_downcast (ref_store,REF_STORE_READ ,"fsck" ); + +return refs->packed_ref_store->be->fsck (refs->packed_ref_store, o); +} +struct ref_storage_be refs_be_files = { .name ="files" , .init = files_ref_store_init,
@@ -3434,5 +3443,7 @@ struct ref_storage_be refs_be_files = { .reflog_exists = files_reflog_exists, .create_reflog = files_create_reflog, .delete_reflog = files_delete_reflog, - .reflog_expire = files_reflog_expire + .reflog_expire = files_reflog_expire, + + .fsck = files_fsck, };
refs/packed-backend.c +8 -0
@@ -1735,6 +1735,12 @@ static struct ref_iterator *packed_reflog_iterator_begin(struct ref_store *ref_sreturn empty_ref_iterator_begin (); } +static int packed_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_packed = { .name ="packed" , .init = packed_ref_store_init,
@@ -1762,4 +1768,6 @@ struct ref_storage_be refs_be_packed = { .create_reflog = NULL, .delete_reflog = NULL, .reflog_expire = NULL, + + .fsck = packed_fsck, };
refs/refs-internal.h +6 -0
@@ -4,6 +4,7 @@#include "refs.h" #include "iterator.h" +struct fsck_options ;struct ref_transaction ;/*
@@ -650,6 +651,9 @@ typedef int read_raw_ref_fn(struct ref_store *ref_store, const char *refname,typedef int read_symbolic_ref_fn (struct ref_store *ref_store,const char *refname,struct strbuf *referent); +typedef int fsck_fn (struct ref_store *ref_store, +struct fsck_options *o); +struct ref_storage_be {const char *name;ref_store_init_fn *init;
@@ -677,6 +681,8 @@ struct ref_storage_be {create_reflog_fn *create_reflog;delete_reflog_fn *delete_reflog;reflog_expire_fn *reflog_expire; + +fsck_fn *fsck; };extern struct ref_storage_be refs_be_files;
refs/reftable-backend.c +8 -0
@@ -2303,6 +2303,12 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store,return ret; } +static int reftable_be_fsck (struct ref_store *ref_store, +struct fsck_options *o) +{ +return 0 ; +} +struct ref_storage_be refs_be_reftable = { .name ="reftable" , .init = reftable_be_init,
@@ -2330,4 +2336,6 @@ struct ref_storage_be refs_be_reftable = { .create_reflog = reftable_be_create_reflog, .delete_reflog = reftable_be_delete_reflog, .reflog_expire = reftable_be_reflog_expire, + + .fsck = reftable_be_fsck, };
Signature · 1 line
2.45.2
PATCH v14 8/11 builtin/refs: add verify subcommand
In reply to shejialuoIntroduce a new subcommand "verify" in git-refs(1) to allow the user to check the reference database consistency and also this subcommand will be used as the entry point of checking refs for "git-fsck(1)".
Add "verbose" field into "fsck_options" to indicate whether we should print verbose messages when checking refs and objects consistency.
Remove bit-field for "strict" field, this is because we cannot take address of a bit-field which makes it unhandy to set member variables when parsing the command line options.
The "git-fsck(1)" declares "fsck_options" variable with "static" identifier which avoids complaint by the leak-checker. However, in "git-refs verify", we need to do memory clean manually. Thus add "fsck_options_clear" function in "fsck.c" to provide memory clean operation.
--- Documentation/git-refs.txt | 13 +++++++++++++ builtin/refs.c | 34 ++++++++++++++++++++++++++++++++++ fsck.c | 11 +++++++++++ fsck.h | 8 +++++++- 4 files changed, 65 insertions(+), 1 deletion(-)
Documentation/git-refs.txt +13 -0
@@ -10,6 +10,7 @@ SYNOPSIS -------- [verse] 'git refs migrate' --ref-format=<format> [--dry-run] +'git refs verify' [--strict] [--verbose] DESCRIPTION -----------
@@ -22,6 +23,9 @@ COMMANDS migrate:: Migrate ref store between different formats. +verify:: + Verify reference database consistency. + OPTIONS -------
@@ -39,6 +43,15 @@ include::ref-storage-format.txt[] can be used to double check that the migration works as expected before performing the actual migration. +The following options are specific to 'git refs verify': + +--strict:: + Enable stricter error checking. This will cause warnings to be + reported as errors. See linkgit:git-fsck[1]. + +--verbose:: + When verifying the reference database consistency, be chatty. + KNOWN LIMITATIONS -----------------
builtin/refs.c +34 -0
@@ -1,4 +1,6 @@#include "builtin.h" +#include "config.h" +#include "fsck.h" #include "parse-options.h" #include "refs.h" #include "repository.h"
@@ -7,6 +9,9 @@#define REFS_MIGRATE_USAGE \ N_("git refs migrate --ref-format=<format> [--dry-run]") +#define REFS_VERIFY_USAGE \ + N_("git refs verify [--strict] [--verbose]") +static int cmd_refs_migrate (int argc,const char **argv,const char *prefix) {const char *const migrate_usage[] = {
@@ -58,15 +63,44 @@ static int cmd_refs_migrate(int argc, const char **argv, const char *prefix)return err; } +static int cmd_refs_verify (int argc,const char **argv,const char *prefix) +{ +struct fsck_options fsck_refs_options =FSCK_REFS_OPTIONS_DEFAULT ; +const char *const verify_usage[] = { +REFS_VERIFY_USAGE , +NULL , + }; +struct option options[] = { +OPT_BOOL (0 ,"verbose" , &fsck_refs_options.verbose,N_ ("be verbose" )), +OPT_BOOL (0 ,"strict" , &fsck_refs_options.strict,N_ ("enable strict checking" )), +OPT_END (), + }; +int ret; + + argc =parse_options (argc, argv, prefix, options, verify_usage,0 ); +if (argc) +usage (_ ("'git refs verify' takes no arguments" )); + +git_config (git_fsck_config, &fsck_refs_options); +prepare_repo_settings (the_repository); + + ret =refs_fsck (get_main_ref_store (the_repository), &fsck_refs_options); + +fsck_options_clear (&fsck_refs_options); +return ret; +} +int cmd_refs (int argc,const char **argv,const char *prefix) {const char *const refs_usage[] = {REFS_MIGRATE_USAGE , +REFS_VERIFY_USAGE ,NULL , };parse_opt_subcommand_fn *fn =NULL ;struct option opts[] = {OPT_SUBCOMMAND ("migrate" , &fn, cmd_refs_migrate), +OPT_SUBCOMMAND ("verify" , &fn, cmd_refs_verify),OPT_END (), };
fsck.c +11 -0
@@ -1336,6 +1336,17 @@ int fsck_finish(struct fsck_options *options)return ret; } +void fsck_options_clear (struct fsck_options *options) +{ +free (options->msg_type); +oidset_clear (&options->skip_oids); +oidset_clear (&options->gitmodules_found); +oidset_clear (&options->gitmodules_done); +oidset_clear (&options->gitattributes_found); +oidset_clear (&options->gitattributes_done); +kh_clear_oid_map (options->object_names); +} +int git_fsck_config (const char *var,const char *value,const struct config_context *ctx,void *cb) {
fsck.h +7 -1
@@ -153,7 +153,8 @@ struct fsck_ref_report {struct fsck_options {fsck_walk_func walk;fsck_error error_func; -unsigned strict:1 ; +unsigned strict; +unsigned verbose;enum fsck_msg_type *msg_type;struct oidset skip_oids;struct oidset gitmodules_found;
@@ -231,6 +232,11 @@ int fsck_tag_standalone(const struct object_id *oid, const char *buffer,*/ int fsck_finish (struct fsck_options *options); +/* +* Clear the fsck_options struct, freeing any allocated memory. +*/ +void fsck_options_clear (struct fsck_options *options); +/* * Report an error or warning for refs. */
Signature · 1 line
2.45.2
PATCH v14 9/11 builtin/fsck: add `git-refs verify` child process
In reply to shejialuoIntroduce a new function "fsck_refs" that initializes and runs a child process to execute the "git-refs verify" command.
--- builtin/fsck.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
builtin/fsck.c +17 -0
@@ -899,6 +899,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)return res; } +static void fsck_refs (void ) +{ +struct child_process refs_verify =CHILD_PROCESS_INIT ; +child_process_init (&refs_verify); + refs_verify.git_cmd =1 ; +strvec_pushl (&refs_verify.args,"refs" ,"verify" ,NULL ); +if (verbose) +strvec_push (&refs_verify.args,"--verbose" ); +if (check_strict) +strvec_push (&refs_verify.args,"--strict" ); + +if (run_command (&refs_verify)) + errors_found |=ERROR_REFS ; +} +static char const *const fsck_usage[] = { N_("git fsck [--tags] [--root] [--unreachable] [--cache] [--no-reflogs]\n" " [--[no-]full] [--strict] [--verbose] [--lost-found]\n"
@@ -1068,6 +1083,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)check_connectivity (); +fsck_refs (); +if (the_repository->settings.core_commit_graph) {struct child_process commit_graph_verify =CHILD_PROCESS_INIT ;
Signature · 1 line
2.45.2
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
Quoted 2 lines · shejialuo
Introduce a new function "
fsck_refs" that initializes and runs a child process to execute the "git-refs verify" command.
It's git refs verify, not git-refs verify both in the commit body and subject.
Quoted 19 lines
Mentored-by Patrick Steinhardt <[email protected]>Mentored-by Karthik Nayak <[email protected]>Signed-off-by shejialuo <[email protected]>--- builtin/fsck.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)builtin/fsck.c +4
-0@@ -899,6 +899,21 @@ static int check_pack_rev_indexes(struct repository *r, int show_progress)return res; } +static void fsck_refs (void ) +{ +struct child_process refs_verify =CHILD_PROCESS_INIT ; +child_process_init (&refs_verify);
builtin/fsck.c static int check_pack_rev_indexes(struct repository *r, int show_progress) · line 915
Quoted 10 lines
builtin/fsck.c +10
-0+ refs_verify.git_cmd =1 ; +strvec_pushl (&refs_verify.args,"refs" ,"verify" ,NULL ); +if (verbose) +strvec_push (&refs_verify.args,"--verbose" ); +if (check_strict) +strvec_push (&refs_verify.args,"--strict" ); + +if (run_command (&refs_verify)) + errors_found |=ERROR_REFS ; +}
Okay. I think that it's sensible to execute this as part of git-fsck(1). But do we want to provide an option to disable this new check, as well?
It does feel a bit like opening a can of worms, though. None of the other checks have trivial ways to disable them, and git-fsck(1) is gaining more and more checks. So if we can disable ref checks, we may also want to have options to disable checks for objects, connectivity, reverse indices, indices, commit graphs and whatnot. In other words, in my opinion we need to think a bit bigger and design a proper UI around this.
But I don't think that should happen as part of this commit series, as it is already big enough. So either we just accept this patch as-is. Or we evict it from this series and handle it in the future together with all the other taks that one may potentially want to disable.
I'd rather pick option two.
Patrick
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
Quoted 26 lines
Quoted 4 lines
builtin/fsck.c +4
-0+static void fsck_refs (void ) +{ +struct child_process refs_verify =CHILD_PROCESS_INIT ; +child_process_init (&refs_verify);builtin/fsck.c static int check_pack_rev_indexes(struct repository *r, int show_progress) · line 915
Quoted 10 lines
builtin/fsck.c +10
-0+ refs_verify.git_cmd =1 ; +strvec_pushl (&refs_verify.args,"refs" ,"verify" ,NULL ); +if (verbose) +strvec_push (&refs_verify.args,"--verbose" ); +if (check_strict) +strvec_push (&refs_verify.args,"--strict" ); + +if (run_command (&refs_verify)) + errors_found |=ERROR_REFS ; +}Okay. I think that it's sensible to execute this as part of git-fsck(1). But do we want to provide an option to disable this new check, as well?
It does feel a bit like opening a can of worms, though. None of the other checks have trivial ways to disable them, and git-fsck(1) is gaining more and more checks. So if we can disable ref checks, we may also want to have options to disable checks for objects, connectivity, reverse indices, indices, commit graphs and whatnot. In other words, in my opinion we need to think a bit bigger and design a proper UI around this.
Quoted 7 lines
But I don't think that should happen as part of this commit series, as it is already big enough. So either we just accept this patch as-is. Or we evict it from this series and handle it in the future together with all the other taks that one may potentially want to disable.
I'd rather pick option two.
After talking with Patrick offline, we decide to drop this patch. At current, we should put this change slowly for the user. Because many people use "git-fsck(1)", currently we don't have a way to disable ref checks by default. It's a little beyond this series.
We may consider later.
Quoted 1 line
Patrick
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
Quoted 34 lines · shejialuo
Quoted 26 lines
Quoted 4 lines
builtin/fsck.c +4
-0+static void fsck_refs (void ) +{ +struct child_process refs_verify =CHILD_PROCESS_INIT ; +child_process_init (&refs_verify);builtin/fsck.c static int check_pack_rev_indexes(struct repository *r, int show_progress) · line 915
Quoted 10 lines
builtin/fsck.c +10
-0+ refs_verify.git_cmd =1 ; +strvec_pushl (&refs_verify.args,"refs" ,"verify" ,NULL ); +if (verbose) +strvec_push (&refs_verify.args,"--verbose" ); +if (check_strict) +strvec_push (&refs_verify.args,"--strict" ); + +if (run_command (&refs_verify)) + errors_found |=ERROR_REFS ; +}Okay. I think that it's sensible to execute this as part of git-fsck(1). But do we want to provide an option to disable this new check, as well?
It does feel a bit like opening a can of worms, though. None of the other checks have trivial ways to disable them, and git-fsck(1) is gaining more and more checks. So if we can disable ref checks, we may also want to have options to disable checks for objects, connectivity, reverse indices, indices, commit graphs and whatnot. In other words, in my opinion we need to think a bit bigger and design a proper UI around this.
Quoted 7 lines
But I don't think that should happen as part of this commit series, as it is already big enough. So either we just accept this patch as-is. Or we evict it from this series and handle it in the future together with all the other taks that one may potentially want to disable.
I'd rather pick option two.
Quoted 6 lines
After talking with Patrick offline, we decide to drop this patch. At current, we should put this change slowly for the user. Because many people use "git-fsck(1)", currently we don't have a way to disable ref checks by default. It's a little beyond this series.
We may consider later.
Hmph, I am fine with the approach to take it slower.
BUT.
Here is what the diffstat for the whole thing in the updated round v15 looked like. Do most of these changes outside refs/ still needed if we do not give any way to even optionally enable it via "fsck" for those who feel adventurous? Should we still be touching fsck.[ch] and other fsck related infrastructure in the series?
Documentation/fsck-msgids.txt | 6 ++ Documentation/git-refs.txt | 13 +++++ builtin/fsck.c | 17 +++--- builtin/mktag.c | 3 +- builtin/refs.c | 34 +++++++++++ fsck.c | 127 +++++++++++++++++++++++++++++++++--------- fsck.h | 76 +++++++++++++++++++------ object-file.c | 9 ++- refs.c | 5 ++ refs.h | 8 +++ refs/debug.c | 11 ++++ refs/files-backend.c | 116 +++++++++++++++++++++++++++++++++++++- refs/packed-backend.c | 8 +++ refs/refs-internal.h | 6 ++ refs/reftable-backend.c | 8 +++ t/t0602-reffiles-fsck.sh | 92 ++++++++++++++++++++++++++++++ 16 files changed, 480 insertions(+), 59 deletions(-)
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
Quoted 11 lines
Hmph, I am fine with the approach to take it slower.
BUT.
Here is what the diffstat for the whole thing in the updated round v15 looked like. Do most of these changes outside refs/ still needed if we do not give any way to even optionally enable it via "fsck" for those who feel adventurous? Should we still be touching fsck.[ch] and other fsck related infrastructure in the series?
From my current standing, regardless of whether we run the "git refs verify" subprocess inside "git-fsck(1)", we must change the fsck infrastructure illustrated by the following:
From the development of this series, we can know the main problem is that fsck error message is highly coupled with the object checks. Even if we don't perform "git refs verify" in "git-fsck(1)", we still need to follow the fsck msg framework when executing "git refs verify". We cannot avoid this.
The content we change for "fsck.[ch]" is mainly the fsck msg part. We do not change anything about the objects.
I agree with you that it would be strange if we do not expose any interfaces for user who are adventurous. Actually we may simply add an option "--refs-experimental" or simply "--refs" to allow the users check ref consistency by using "git-fsck(1)".
I guess the concern that Patrick cares about is that we ONLY make refs optional here, but do not provide options for other checks. It will be strange from this perspective.
Quoted 17 lines
Documentation/fsck-msgids.txt | 6 ++ Documentation/git-refs.txt | 13 +++++ builtin/fsck.c | 17 +++--- builtin/mktag.c | 3 +- builtin/refs.c | 34 +++++++++++ fsck.c | 127 +++++++++++++++++++++++++++++++++--------- fsck.h | 76 +++++++++++++++++++------ object-file.c | 9 ++- refs.c | 5 ++ refs.h | 8 +++ refs/debug.c | 11 ++++ refs/files-backend.c | 116 +++++++++++++++++++++++++++++++++++++- refs/packed-backend.c | 8 +++ refs/refs-internal.h | 6 ++ refs/reftable-backend.c | 8 +++ t/t0602-reffiles-fsck.sh | 92 ++++++++++++++++++++++++++++++ 16 files changed, 480 insertions(+), 59 deletions(-)
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
Quoted 8 lines · shejialuo
I agree with you that it would be strange if we do not expose any interfaces for user who are adventurous. Actually we may simply add an option "--refs-experimental" or simply "--refs" to allow the users check ref consistency by using "git-fsck(1)".
I guess the concern that Patrick cares about is that we ONLY make refs optional here, but do not provide options for other checks. It will be strange from this perspective.
I do not care about strange all that much. I however care about new complexity in the code, complexity that is not taken advantage of and is not exercised.
You said
Quoted 2 lines
From the development of this series, we can know the main problem is that fsck error message is highly coupled with the object checks.
and even if it is true and we have problem in fsck code paths, we cannot see if _your_ solution to that problem is a good one without having the code that exercises your additional code.
But if "git refs verify" does exercise all the new code paths (and the refactored code that existed before this series, sitting now in different places), then I do not have to worry about it. My question was primarily to extract "even though we do not wire this up to fsck, we already have another code paths that uses all these changes" out of you.
Thanks.
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
Quoted 24 lines · Junio C Hamano
Quoted 8 lines · shejialuo
I agree with you that it would be strange if we do not expose any interfaces for user who are adventurous. Actually we may simply add an option "--refs-experimental" or simply "--refs" to allow the users check ref consistency by using "git-fsck(1)".
I guess the concern that Patrick cares about is that we ONLY make refs optional here, but do not provide options for other checks. It will be strange from this perspective.
I do not care about strange all that much. I however care about new complexity in the code, complexity that is not taken advantage of and is not exercised.
You said
Quoted 2 lines
From the development of this series, we can know the main problem is that fsck error message is highly coupled with the object checks.
and even if it is true and we have problem in fsck code paths, we cannot see if _your_ solution to that problem is a good one without having the code that exercises your additional code.
Quoted 7 lines
But if "git refs verify" does exercise all the new code paths (and the refactored code that existed before this series, sitting now in different places), then I do not have to worry about it. My question was primarily to extract "even though we do not wire this up to fsck, we already have another code paths that uses all these changes" out of you.
I understand what you mean here. I can say that "git refs verify" only exercises a part of the new code paths. The main reason why this series changes a lot of "fsck.[ch]" is that I want to expose the "fsck_report_ref" interface to report refs-related errors. So I guess this part should be covered.
At the current implementation, we change the "fsck.[ch]" for the following three things:
1. Refactor "report" to use "fsck_vreport"
2. Create "fsck_report_ref" for refs check.
3. Do some simple renames to distinguish between refs and objects.
We do cover the second case in "git refs verify". But sadly, the first case and third case are covered in "git-fsck(1)". So, "git refs verify" does not exercise the refactored code.
However, I am a little confused. Actually, in the implementation, refs check and objects check are independent.
I think we should wire up "git refs verify" to "git-fsck(1)". Because we have no way to exercise the above case 1 and 3. If we do not, we will bring a lot of complexity here.
Quoted 1 line
Thanks.
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
Quoted 50 lines · shejialuo
Quoted 24 lines · Junio C Hamano
Quoted 8 lines · shejialuo
I agree with you that it would be strange if we do not expose any interfaces for user who are adventurous. Actually we may simply add an option "--refs-experimental" or simply "--refs" to allow the users check ref consistency by using "git-fsck(1)".
I guess the concern that Patrick cares about is that we ONLY make refs optional here, but do not provide options for other checks. It will be strange from this perspective.
I do not care about strange all that much. I however care about new complexity in the code, complexity that is not taken advantage of and is not exercised.
You said
Quoted 2 lines
From the development of this series, we can know the main problem is that fsck error message is highly coupled with the object checks.
and even if it is true and we have problem in fsck code paths, we cannot see if _your_ solution to that problem is a good one without having the code that exercises your additional code.
Quoted 7 lines
But if "git refs verify" does exercise all the new code paths (and the refactored code that existed before this series, sitting now in different places), then I do not have to worry about it. My question was primarily to extract "even though we do not wire this up to fsck, we already have another code paths that uses all these changes" out of you.
I understand what you mean here. I can say that "git refs verify" only exercises a part of the new code paths. The main reason why this series changes a lot of "fsck.[ch]" is that I want to expose the "
fsck_report_ref" interface to report refs-related errors. So I guess this part should be covered.At the current implementation, we change the "fsck.[ch]" for the following three things:
1. Refactor "report" to use "
fsck_vreport"
2. Create "fsck_report_ref" for refs check.
3. Do some simple renames to distinguish between refs and objects.We do cover the second case in "git refs verify". But sadly, the first case and third case are covered in "git-fsck(1)". So, "git refs verify" does not exercise the refactored code.
Quoted 6 lines
However, I am a little confused. Actually, in the implementation, refs check and objects check are independent.
I think we should wire up "git refs verify" to "git-fsck(1)". Because we have no way to exercise the above case 1 and 3. If we do not, we will bring a lot of complexity here.
I don't think that is necessary. Basically, what you are saying is that we seem to be testing only the new refs-related reporting that you have introduced via your refactorings, but not the preexisting objects related functionality.
That isn't true though. The object-specific reporting functions that you have refactored already had callers before, and it still has callers now because you adapted those accordingly. You can assume that those callers already had tests before your refactorings, and as no tests broken you can be reasonably sure that your refactorings are sound for the object related code.
Furthermore, you do exercise the new ref-related parts via a couple of tests that exercise git refs verify. Consequently, all parts of the refactoring are covered by either old or new tests, and we should be good here.
So even though we do not exercise the ref-related parts via git-fsck(1), it is still subjected to tests. Eventually, when we start calling git refs verify in git-fsck(1), we'd introduce more tests that verify that the integration of those two commands works as expected, as well.
So, to summarize: the refactored functionality is both used and tested and I think it's sensible to defer the integration of git-fsck(1) and git-refs(1).
Thanks!
Patrick
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
Quoted 10 lines · Patrick Steinhardt
Quoted 7 lines
Quoted 6 lines
But if "git refs verify" does exercise all the new code paths (and the refactored code that existed before this series, sitting now in different places), then I do not have to worry about it. My question was primarily to extract "even though we do not wire this up to fsck, we already have another code paths that uses all these changes" out of you.
...
So, to summarize: the refactored functionality is both used and tested and I think it's sensible to defer the integration of git-fsck(1) and git-refs(1).
After refactoring, existing functionality about objects are used, of course, (there is no other code that does so), the refactoring lets the code to learn to perform checks on references, and these new checks are exercised by "git refs verify".
I took what shejialuo said that way, and that is fine by me when I said the above ;-). So I think we all are on the same page?
Thanks.
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
Quoted 13 lines · Junio C Hamano
Quoted 10 lines · Patrick Steinhardt
Quoted 7 lines
Quoted 6 lines
But if "git refs verify" does exercise all the new code paths (and the refactored code that existed before this series, sitting now in different places), then I do not have to worry about it. My question was primarily to extract "even though we do not wire this up to fsck, we already have another code paths that uses all these changes" out of you.
...
So, to summarize: the refactored functionality is both used and tested and I think it's sensible to defer the integration of git-fsck(1) and git-refs(1).
Quoted 7 lines
After refactoring, existing functionality about objects are used, of course, (there is no other code that does so), the refactoring lets the code to learn to perform checks on references, and these new checks are exercised by "git refs verify".
I took what shejialuo said that way, and that is fine by me when I said the above ;-). So I think we all are on the same page?
Yup, we are. I was mostly aiming to reassure Jialuo, who was a bit uncertain whether his answers had been sufficient or not.
Thanks!
Patrick
Re: [GSoC][PATCH v14 09/11] builtin/fsck: add `git-refs verify` child process
In reply to Junio C HamanoQuoted 24 lines · Junio C Hamano
Quoted 8 lines · shejialuo
I agree with you that it would be strange if we do not expose any interfaces for user who are adventurous. Actually we may simply add an option "--refs-experimental" or simply "--refs" to allow the users check ref consistency by using "git-fsck(1)".
I guess the concern that Patrick cares about is that we ONLY make refs optional here, but do not provide options for other checks. It will be strange from this perspective.
I do not care about strange all that much. I however care about new complexity in the code, complexity that is not taken advantage of and is not exercised.
You said
Quoted 2 lines
From the development of this series, we can know the main problem is that fsck error message is highly coupled with the object checks.
and even if it is true and we have problem in fsck code paths, we cannot see if _your_ solution to that problem is a good one without having the code that exercises your additional code.
Quoted 6 lines
But if "git refs verify" does exercise all the new code paths (and the refactored code that existed before this series, sitting now in different places), then I do not have to worry about it. My question was primarily to extract "even though we do not wire this up to fsck, we already have another code paths that uses all these changes" out of you.
Yeah, there is quite some complexity introduced in fsck code, where most of the complexity comes from making the reportin functions more generic. And we do end up using that in this series even if we don't integrate git refs verify with git fsck. It brings us the ability to configure the checks performed by git refs verify in the same way like we can configure all the other checks in git fsck, such that we can enable, disable or change severity levels for each of the messages reported by it.
I think that this is a sensible way to go about it, and it leaves us with a more flexible and consistent error reporting infrastructure that we can eventually also use for other commands that git-fsck(1) shells out to.
Patrick