Introduce `refStorage` extension

Hi,

this is the third version of my patch series that introduces the new refStorage extension. This extension will be used for the upcoming reftable backend.

Changes compared to v3:

- The ref_storage_format is now tracked as an unsigned int, proposed by Junio.

- Reworded the commit message in patch 2, proposed by Eric.

- Added a NEEDSWORK comment to get_worktrees_internal(), propose by Eric.

Thanks for your reviews!

Patrick

Patrick Steinhardt (12):
t: introduce DEFAULT_REPO_FORMAT prereq
worktree: skip reading HEAD when repairing worktrees
refs: refactor logic to look up storage backends
setup: start tracking ref storage format
setup: set repository's formats on init
setup: introduce "extensions.refStorage" extension
setup: introduce GIT_DEFAULT_REF_FORMAT envvar
t: introduce GIT_TEST_DEFAULT_REF_FORMAT envvar builtin/rev-parse: introduce --show-ref-format flag builtin/init: introduce --ref-format= value flag builtin/clone: introduce --ref-format= value flag
t9500: write "extensions.refstorage" into config

 Documentation/config/extensions.txt           | 11 +++
 Documentation/git-clone.txt                   |  6 ++
 Documentation/git-init.txt                    |  7 ++
 Documentation/git-rev-parse.txt               |  3 +
 Documentation/git.txt                         |  5 ++
 Documentation/ref-storage-format.txt          |  1 +
 .../technical/repository-version.txt          |  5 ++
 builtin/clone.c                               | 17 ++++-
 builtin/init-db.c                             | 15 +++-
 builtin/rev-parse.c                           |  4 ++
 refs.c                                        | 34 ++++++---
 refs.h                                        |  3 +
 refs/debug.c                                  |  1 -
 refs/files-backend.c                          |  1 -
 refs/packed-backend.c                         |  1 -
 refs/refs-internal.h                          |  1 -
 repository.c                                  |  6 ++
 repository.h                                  |  7 ++
 setup.c                                       | 66 +++++++++++++++--
 setup.h                                       | 10 ++-
 t/README                                      |  3 +
 t/t0001-init.sh                               | 70 +++++++++++++++++++
 t/t1500-rev-parse.sh                          | 17 +++++
 t/t3200-branch.sh                             |  2 +-
 t/t5601-clone.sh                              | 17 +++++
 t/t9500-gitweb-standalone-no-errors.sh        |  5 ++
 t/test-lib-functions.sh                       |  5 ++
 t/test-lib.sh                                 | 15 +++-
 worktree.c                                    | 31 +++++---
 29 files changed, 334 insertions(+), 35 deletions(-)
 create mode 100644 Documentation/ref-storage-format.txt

Range-diff against v2:

 1:  3613439cb7 =  1:  578deaabcf t: introduce DEFAULT_REPO_FORMAT prereq
 2:  ecf4f1ddee !  2:  77c7213c66 worktree: skip reading HEAD when repairing worktrees

@@ Commit message
is logic that we resolve their respective worktree HEADs, even though that information may not actually be needed in the end by all callers.

- In the context of git-init(1) this is about to become a problem, because
- we do not have a repository that was set up via setup_git_directory()
- or friends. Consequentially, it is not yet fully initialized at the time
- of calling repair_worktrees(), and properly setting up all parts of
- the repository in init_db() before we repair worktrees is not an easy
- thing to do. While this is okay right now where we only have a single
- reference backend in Git, once we gain a second one we would be trying
- to look up the worktree HEADs before we have figured out the reference
- format, which does not work.
+ Although not a problem presently with the file-based reference backend,
+ it will become a problem with the upcoming reftable backend. In the
+ context of git-init(1) we do not have a fully-initialized repository set
+ up via setup_git_directory() or friends. Consequently, we do not know
+ about the repository format when repair_worktrees() is called, and
+ properly setting up all parts of the repositroy in init_db() before we
+ try to repair worktrees is not an easy task. With the introduction of
+ the reftable backend, we would ultimately try to look up the worktree
+ HEADs before we have figured out the reference format, which does not
+ work.

We do not require the worktree HEADs at all to repair worktrees. So let's fix this issue by skipping over the step that reads them.
@@ worktree.c: static void mark_current_worktree(struct worktree **worktrees)
}

-struct worktree **get_worktrees(void)
++/*
++ * NEEDSWORK: This function exists so that we can look up metadata of a ++ * worktree without trying to access any of its internals like the refdb. It ++ * would be preferable to instead have a corruption-tolerant function for ++ * retrieving worktree metadata that could be used when the worktree is known ++ * to not be in a healthy state, e.g. when creating or repairing it. ++ */
+static struct worktree **get_worktrees_internal(int skip_reading_head)
{
struct worktree **list = NULL;
3: 12329b99b7 ! 3: 47649570bf refs: refactor logic to look up storage backends

    @@ refs.c
     +};
      
     -static struct ref_storage_be *find_ref_storage_backend(const char *name)
    -+static const struct ref_storage_be *find_ref_storage_backend(int ref_storage_format)
    ++static const struct ref_storage_be *find_ref_storage_backend(unsigned int ref_storage_format)
      {
     -	struct ref_storage_be *be;
     -	for (be = refs_backends; be; be = be->next)
    @@ refs.c
      	return NULL;
      }
      
    -+int ref_storage_format_by_name(const char *name)
    ++unsigned int ref_storage_format_by_name(const char *name)
     +{
    -+	for (int i = 0; i < ARRAY_SIZE(refs_backends); i++)
    ++	for (unsigned int i = 0; i < ARRAY_SIZE(refs_backends); i++)
     +		if (refs_backends[i] && !strcmp(refs_backends[i]->name, name))
     +			return i;
     +	return REF_STORAGE_FORMAT_UNKNOWN;
     +}
     +
    -+const char *ref_storage_format_to_name(int ref_storage_format)
    ++const char *ref_storage_format_to_name(unsigned int ref_storage_format)
     +{
     +	const struct ref_storage_be *be = find_ref_storage_backend(ref_storage_format);
     +	if (!be)
    @@ refs.c: static struct ref_store *ref_store_init(struct repository *repo,
      {
     -	const char *be_name = "files";
     -	struct ref_storage_be *be = find_ref_storage_backend(be_name);
    -+	int format = REF_STORAGE_FORMAT_FILES;
    ++	unsigned int format = REF_STORAGE_FORMAT_FILES;
     +	const struct ref_storage_be *be = find_ref_storage_backend(format);
      	struct ref_store *refs;
      
    @@ refs.h: struct string_list;
      struct string_list_item;
      struct worktree;
      
    -+int ref_storage_format_by_name(const char *name);
    -+const char *ref_storage_format_to_name(int ref_storage_format);
    ++unsigned int ref_storage_format_by_name(const char *name);
    ++const char *ref_storage_format_to_name(unsigned int ref_storage_format);
     +
      /*
       * Resolve a reference, recursively following symbolic refererences.
 4:  ddd099fbaf !  4:  837764d0b5 setup: start tracking ref storage format
    @@ refs.c: static struct ref_store *ref_store_init(struct repository *repo,
      					const char *gitdir,
      					unsigned int flags)
      {
    --	int format = REF_STORAGE_FORMAT_FILES;
    +-	unsigned int format = REF_STORAGE_FORMAT_FILES;
     -	const struct ref_storage_be *be = find_ref_storage_backend(format);
     +	const struct ref_storage_be *be;
      	struct ref_store *refs;
    @@ repository.c: void repo_set_hash_algo(struct repository *repo, int hash_algo)
      	repo->hash_algo = &hash_algos[hash_algo];
      }
      
    -+void repo_set_ref_storage_format(struct repository *repo, int format)
    ++void repo_set_ref_storage_format(struct repository *repo, unsigned int format)
     +{
     +	repo->ref_storage_format = format;
     +}
    @@ repository.h: struct repository {
      	const struct git_hash_algo *hash_algo;
      
     +	/* Repository's reference storage format, as serialized on disk. */
    -+	int ref_storage_format;
    ++	unsigned int ref_storage_format;
     +
      	/* A unique-id for tracing purposes. */
      	int trace2_repo_id;
    @@ repository.h: void repo_set_gitdir(struct repository *repo, const char *root,
      		     const struct set_gitdir_args *extra_args);
      void repo_set_worktree(struct repository *repo, const char *path);
      void repo_set_hash_algo(struct repository *repo, int algo);
    -+void repo_set_ref_storage_format(struct repository *repo, int format);
    ++void repo_set_ref_storage_format(struct repository *repo, unsigned int format);
      void initialize_the_repository(void);
      RESULT_MUST_BE_USED
      int repo_init(struct repository *r, const char *gitdir, const char *worktree);
    @@ setup.c: static int is_reinit(void)
      }
      
     -void create_reference_database(const char *initial_branch, int quiet)
    -+void create_reference_database(int ref_storage_format,
    ++void create_reference_database(unsigned int ref_storage_format,
     +			       const char *initial_branch, int quiet)
      {
      	struct strbuf err = STRBUF_INIT;
    @@ setup.c: static void validate_hash_algorithm(struct repository_format *repo_fmt,
      	}
      }
      
    -+static void validate_ref_storage_format(struct repository_format *repo_fmt, int format)
    ++static void validate_ref_storage_format(struct repository_format *repo_fmt,
    ++					unsigned int format)
     +{
     +	if (repo_fmt->version >= 0 &&
     +	    format != REF_STORAGE_FORMAT_UNKNOWN &&
    @@ setup.c: static void validate_hash_algorithm(struct repository_format *repo_fmt,
      int init_db(const char *git_dir, const char *real_git_dir,
     -	    const char *template_dir, int hash, const char *initial_branch,
     +	    const char *template_dir, int hash,
    -+	    int ref_storage_format, const char *initial_branch,
    ++	    unsigned int ref_storage_format,
    ++	    const char *initial_branch,
      	    int init_shared_repository, unsigned int flags)
      {
      	int reinit;
    @@ setup.h: struct repository_format {
      	int worktree_config;
      	int is_bare;
      	int hash_algo;
    -+	int ref_storage_format;
    ++	unsigned int ref_storage_format;
      	int sparse_index;
      	char *work_tree;
      	struct string_list unknown_extensions;
    @@ setup.h: void check_repository_format(struct repository_format *fmt);
      
      int init_db(const char *git_dir, const char *real_git_dir,
      	    const char *template_dir, int hash_algo,
    -+	    int ref_storage_format,
    ++	    unsigned int ref_storage_format,
      	    const char *initial_branch, int init_shared_repository,
      	    unsigned int flags);
      void initialize_repository_version(int hash_algo, int reinit);
     -void create_reference_database(const char *initial_branch, int quiet);
    -+void create_reference_database(int ref_storage_format,
    ++void create_reference_database(unsigned int ref_storage_format,
     +			       const char *initial_branch, int quiet);
      
      /*
 5:  01a1e58a97 =  5:  a51da56d9b setup: set repository's formats on init
 6:  0a586fa648 !  6:  a1e03e4392 setup: introduce "extensions.refStorage" extension
    @@ setup.c: static enum extension_result handle_extension(const char *var,
      		data->hash_algo = format;
      		return EXTENSION_OK;
     +	} else if (!strcmp(ext, "refstorage")) {
    -+		int format;
    ++		unsigned int format;
     +
     +		if (!value)
     +			return config_error_nonbool(var);
    @@ setup.c: static int needs_work_tree_config(const char *git_dir, const char *work
      }
      
     -void initialize_repository_version(int hash_algo, int reinit)
    -+void initialize_repository_version(int hash_algo, int ref_storage_format,
    ++void initialize_repository_version(int hash_algo,
    ++				   unsigned int ref_storage_format,
     +				   int reinit)
      {
      	char repo_version_string[10];
    @@ setup.c: static int create_default_files(const char *template_path,
     
      ## setup.h ##
     @@ setup.h: int init_db(const char *git_dir, const char *real_git_dir,
    - 	    int ref_storage_format,
    + 	    unsigned int ref_storage_format,
      	    const char *initial_branch, int init_shared_repository,
      	    unsigned int flags);
     -void initialize_repository_version(int hash_algo, int reinit);
    -+void initialize_repository_version(int hash_algo, int ref_storage_format,
    ++void initialize_repository_version(int hash_algo,
    ++				   unsigned int ref_storage_format,
     +				   int reinit);
    - void create_reference_database(int ref_storage_format,
    + void create_reference_database(unsigned int ref_storage_format,
      			       const char *initial_branch, int quiet);

7: 6d8754f73a ! 7: 5ffc70e9be setup: introduce GIT_DEFAULT_REF_FORMAT envvar
@@ Documentation/git.txt: double-quotes and respecting backslash escapes. E.g., the

## setup.c ##
@@ setup.c: static void validate_hash_algorithm(struct repository_format *repo_fmt, int hash
-
- static void validate_ref_storage_format(struct repository_format *repo_fmt, int format)
+ static void validate_ref_storage_format(struct repository_format *repo_fmt,
+ unsigned int format)
{
+ const char *name = getenv("GIT_DEFAULT_REF_FORMAT");
+

 8:  c645932f3d =  8:  13c074acdf t: introduce GIT_TEST_DEFAULT_REF_FORMAT envvar
 9:  761d647770 =  9:  4ee3c9a2d1 builtin/rev-parse: introduce `--show-ref-format` flag
10:  e382b5bf08 ! 10:  25773e3560 builtin/init: introduce `--ref-format=` value flag
    @@ builtin/init-db.c: int cmd_init_db(int argc, const char **argv, const char *pref
     +	const char *ref_format = NULL;
      	const char *initial_branch = NULL;
      	int hash_algo = GIT_HASH_UNKNOWN;
    -+	int ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN;
    ++	unsigned int ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN;
      	int init_shared_repository = -1;
      	const struct option init_db_options[] = {
      		OPT_STRING(0, "template", &template_dir, N_("template-directory"),
11:  257233658d ! 11:  3f1cb6b9e5 builtin/clone: introduce `--ref-format=` value flag
    @@ builtin/clone.c: int cmd_clone(int argc, const char **argv, const char *prefix)
      	int submodule_progress;
      	int filter_submodules = 0;
      	int hash_algo;
    -+	int ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN;
    ++	unsigned int ref_storage_format = REF_STORAGE_FORMAT_UNKNOWN;
      	const int do_not_override_repo_unix_permissions = -1;
      
      	struct transport_ls_refs_options transport_ls_refs_options =

12: b8cd06ec53 = 12: 2e7682b2f3 t9500: write "extensions.refstorage" into config

base-commit: e79552d19784ee7f4bbce278fe25f93fbda196fa

2.43.GIT