Moving global state into the repository object (part 1)
cover Moving global state into the repository object (part 1)
This is a real take on the first part of the recent RFC[1].
Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories" might be a good breaking point for a first part at that RFC at patch 38. This series is smaller and contains only 26 patches as the patches in the big RFC were slightly out of order.
I developed this series partly by writing patches, but mostly by cherrypicking from that RFC on top of current master. I noticed no external conflicts apart from one addition to the repositories _INIT macro, which was easy to resolve.
Comments in the early range of that RFC were on 003 where Junio pointed out that the coccinelle patch ought to be not in contrib/coccinelle, so I put it in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.
brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself without any suggestion to include into this series[3].
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the ignore_env flag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.
Eric Sunshine brought up memory leaks with the RFC, and I would think to have plugged all holes.
[1] https://public-inbox.org/git/[email protected]/
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/
Thanks,
Stefan
Jonathan Nieder (8):
pack: move prepare_packed_git_run_once to object store
pack: move approximate object count to object store
sha1_file: add repository argument to sha1_file_name
sha1_file: add repository argument to map_sha1_file
sha1_file: allow stat_sha1_file to handle arbitrary repositories
sha1_file: allow open_sha1_file to handle arbitrary repositories
sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
sha1_file: allow sha1_loose_object_info to handle arbitrary
repositories
Stefan Beller (18):
repository: introduce raw object store field
object-store: move alt_odb_list and alt_odb_tail to object store
object-store: free alt_odb_list
object-store: move packed_git and packed_git_mru to object store
object-store: close all packs upon clearing the object store
sha1_file: add raw_object_store argument to alt_odb_usable
sha1_file: add repository argument to link_alt_odb_entry
sha1_file: add repository argument to read_info_alternates
sha1_file: add repository argument to link_alt_odb_entries
sha1_file: add repository argument to prepare_alt_odb
sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
sha1_file: allow prepare_alt_odb to handle arbitrary repositories
sha1_file: add repository argument to stat_sha1_file
sha1_file: add repository argument to open_sha1_file
sha1_file: add repository argument to map_sha1_file_1
sha1_file: add repository argument to sha1_loose_object_info
sha1_file: allow sha1_file_name to handle arbitrary repositories
sha1_file: allow map_sha1_file to handle arbitrary repositories
builtin/am.c | 2 +- builtin/clone.c | 2 +- builtin/count-objects.c | 6 +- builtin/fetch.c | 2 +- builtin/fsck.c | 13 ++- builtin/gc.c | 4 +- builtin/grep.c | 2 +- builtin/index-pack.c | 1 + builtin/merge.c | 2 +- builtin/pack-objects.c | 21 ++-- builtin/pack-redundant.c | 6 +- builtin/receive-pack.c | 3 +- cache.h | 46 ++------ contrib/coccinelle/refactoring/packed_git.cocci | 7 ++ environment.c | 5 +- fast-import.c | 6 +- http-backend.c | 6 +- http-push.c | 1 + http-walker.c | 4 +- http.c | 6 +- mru.h | 1 + object-store.h | 75 +++++++++++++ object.c | 26 +++++ pack-bitmap.c | 4 +- pack-check.c | 1 + pack-revindex.c | 1 + packfile.c | 64 +++++------ packfile.h | 2 +- path.c | 2 +- reachable.c | 1 + repository.c | 17 ++- repository.h | 7 +- server-info.c | 6 +- sha1_file.c | 135 +++++++++++++----------- sha1_name.c | 11 +- streaming.c | 5 +- 36 files changed, 321 insertions(+), 182 deletions(-) create mode 100644 contrib/coccinelle/refactoring/packed_git.cocci create mode 100644 object-store.h
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 1/26 repository: introduce raw object store field
The raw object store field will contain any objects needed for access to objects in a given repository.
This patch introduces the raw object store and populates it with the objectdir, which used to be part of the repository struct.
As the struct gains members, we'll also populate the function to clear the memory for these members.
In a later we'll introduce a struct object_parser, that will complement the object parsing in a repository struct: The raw object parser is the layer that will provide access to raw object content, while the higher level object parser code will parse raw objects and keeps track of parenthood and other object relationships using 'struct object'. For now only add the lower level to the repository struct.
--- builtin/grep.c | 2 +- environment.c | 5 +++-- object-store.h | 15 +++++++++++++++ object.c | 5 +++++ path.c | 2 +- repository.c | 17 +++++++++++++---- repository.h | 7 ++++--- 7 files changed, 42 insertions(+), 11 deletions(-) create mode 100644 object-store.h
builtin/grep.c +1 -1
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,* object. */ grep_read_lock (); -add_to_alternates_memory (submodule.objectdir); +add_to_alternates_memory (submodule.objects.objectdir);grep_read_unlock ();if (oid) {
environment.c +3 -2
@@ -13,6 +13,7 @@#include "refs.h" #include "fmt-merge-msg.h" #include "commit.h" +#include "object-store.h" int trust_executable_bit =1 ;int trust_ctime =1 ;
@@ -242,9 +243,9 @@ const char *get_git_work_tree(void)char *get_object_directory (void ) { -if (!the_repository->objectdir) +if (!the_repository->objects.objectdir)BUG ("git environment hasn't been setup" ); -return the_repository->objectdir; +return the_repository->objects.objectdir; }int odb_mkstemp (struct strbuf *template,const char *pattern)
object-store.h new file +15 -0
@@ -0,0 +1,15 @@ +#ifndef OBJECT_STORE_H +#define OBJECT_STORE_H + +struct raw_object_store { +/* +* Path to the repository's object store. +* Cannot be NULL after initialization. +*/ +char *objectdir; +}; +#define RAW_OBJECT_STORE_INIT { NULL } + +void raw_object_store_clear (struct raw_object_store *o); + +#endif /* OBJECT_STORE_H */
object.c +5 -0
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags) obj->flags &= ~flags; } } + +void raw_object_store_clear (struct raw_object_store *o) +{ +free (o->objectdir); +}
path.c +1 -1
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,strbuf_splice (buf,0 , buf->len, repo->index_file,strlen (repo->index_file)); else if (dir_prefix(base, "objects")) - replace_dir(buf, git_dir_len + 7, repo->objectdir); + replace_dir(buf, git_dir_len + 7, repo->objects.objectdir); else if (git_hooks_path && dir_prefix(base, "hooks")) replace_dir(buf, git_dir_len + 5, git_hooks_path); else if (repo->different_commondir)
repository.c +13 -4
@@ -1,11 +1,18 @@#include "cache.h" #include "repository.h" +#include "object-store.h" #include "config.h" #include "submodule-config.h" /* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL , +RAW_OBJECT_STORE_INIT , +NULL ,NULL ,NULL , +NULL ,NULL ,NULL , + &the_index, + &hash_algos[GIT_HASH_SHA1 ], +0 ,0 };struct repository *the_repository = &the_repo;
@@ -42,8 +49,8 @@ static void repo_setup_env(struct repository *repo) !repo->ignore_env);free (repo->commondir); repo->commondir =strbuf_detach (&sb,NULL ); -free (repo->objectdir); - repo->objectdir =git_path_from_env (DB_ENVIRONMENT , repo->commondir, +free (repo->objects.objectdir); + repo->objects.objectdir =git_path_from_env (DB_ENVIRONMENT , repo->commondir,"objects" , !repo->ignore_env);free (repo->graft_file); repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
@@ -209,12 +216,14 @@ void repo_clear(struct repository *repo) {FREE_AND_NULL (repo->gitdir);FREE_AND_NULL (repo->commondir); -FREE_AND_NULL (repo->objectdir);FREE_AND_NULL (repo->graft_file);FREE_AND_NULL (repo->index_file);FREE_AND_NULL (repo->worktree);FREE_AND_NULL (repo->submodule_prefix); +raw_object_store_clear (&repo->objects); +memset (&repo->objects,0 ,sizeof (repo->objects)); +if (repo->config) {git_configset_clear (repo->config);FREE_AND_NULL (repo->config);
repository.h +4 -3
@@ -1,6 +1,8 @@#ifndef REPOSITORY_H #define REPOSITORY_H +#include "object-store.h" +struct config_set ;struct index_state ;struct submodule_cache ;
@@ -21,10 +23,9 @@ struct repository {char *commondir;/* -* Path to the repository's object store. -* Cannot be NULL after initialization. +* Holds any information related to the object store. */ -char *objectdir; +struct raw_object_store objects;/* * Path to the repository's graft file.
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 2/26 object-store: move alt_odb_list and alt_odb_tail to object store
In reply to Stefan BellerIn a process with multiple repositories open, alternates should be associated to a single repository and not shared globally. Move alt_odb_list and alt_odb_tail into the_repository and adjust callers to reflect this.
Now that the alternative object data base is per repository, we're leaking its memory upon freeing a repository. The next patch plugs this hole.
No functional change intended.
--- builtin/fsck.c | 4 +++- cache.h | 4 ++-- object-store.h | 7 ++++++- packfile.c | 3 ++- sha1_file.c | 25 ++++++++++++------------- sha1_name.c | 3 ++- 6 files changed, 27 insertions(+), 19 deletions(-)
builtin/fsck.c +3 -1
@@ -1,5 +1,6 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "commit.h" #include "tree.h"
@@ -694,7 +695,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)fsck_object_dir (get_object_directory ());prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; + alt; alt = alt->next)fsck_object_dir (alt->path);if (check_full) {
cache.h +2 -2
@@ -1573,7 +1573,7 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);extern void schedule_dir_for_removal (const char *name,int len);extern void remove_scheduled_dirs (void ); -extern struct alternate_object_database { +struct alternate_object_database {struct alternate_object_database *next;/* see alt_scratch_buf() */
@@ -1591,7 +1591,7 @@ extern struct alternate_object_database {struct oid_array loose_objects_cache;char path[FLEX_ARRAY ]; -} *alt_odb_list; +};extern void prepare_alt_odb (void );extern char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);
object-store.h +6 -1
@@ -1,14 +1,19 @@#ifndef OBJECT_STORE_H #define OBJECT_STORE_H +#include "cache.h" +struct raw_object_store {/* * Path to the repository's object store. * Cannot be NULL after initialization. */ char *objectdir; + +struct alternate_object_database *alt_odb_list; +struct alternate_object_database **alt_odb_tail; }; -#define RAW_OBJECT_STORE_INIT { NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +2 -1
@@ -1,6 +1,7 @@#include "cache.h" #include "mru.h" #include "pack.h" +#include "repository.h" #include "dir.h" #include "mergesort.h" #include "packfile.h"
@@ -880,7 +881,7 @@ void prepare_packed_git(void)return ;prepare_packed_git_one (get_object_directory (),1 );prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();prepare_packed_git_mru ();
sha1_file.c +12 -13
@@ -22,6 +22,7 @@#include "pack-revindex.h" #include "sha1-lookup.h" #include "bulk-checkin.h" +#include "repository.h" #include "streaming.h" #include "dir.h" #include "mru.h"
@@ -346,9 +347,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,return buf->buf; } -struct alternate_object_database *alt_odb_list; -static struct alternate_object_database **alt_odb_tail; -/* * Return non-zero iff the path is usable as an alternate object database. */
@@ -368,7 +366,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)* Prevent the common mistake of listing the same * thing twice, or object directory itself. */ -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {if (!fspathcmp (path->buf, alt->path))return 0 ; }
@@ -428,8 +426,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base, ent =alloc_alt_odb (pathbuf.buf);/* add the alternate entry */ - *alt_odb_tail = ent; - alt_odb_tail = &(ent->next); + *the_repository->objects.alt_odb_tail = ent; + the_repository->objects.alt_odb_tail = &(ent->next); ent->next =NULL ;/* recursively add alternates */
@@ -563,7 +561,7 @@ void add_to_alternates_file(const char *reference)fprintf_or_die (out,"%s\n" , reference);if (commit_lock_file (&lock))die_errno ("unable to move new alternates file into place" ); -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)link_alt_odb_entries (reference,'\n' ,NULL ,0 ); }free (alts);
@@ -661,7 +659,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)int r =0 ;prepare_alt_odb (); -for (ent = alt_odb_list; ent; ent = ent->next) { +for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) { r =fn (ent, cb);if (r)break ;
@@ -673,12 +671,13 @@ void prepare_alt_odb(void) {const char *alt; -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)return ; alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - alt_odb_tail = &alt_odb_list; + the_repository->objects.alt_odb_tail = + &the_repository->objects.alt_odb_list;link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 );read_info_alternates (get_object_directory (),0 );
@@ -717,7 +716,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen) {struct alternate_object_database *alt;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {const char *path =alt_sha1_path (alt, sha1);if (check_and_freshen_file (path, freshen))return 1 ;
@@ -873,7 +872,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,prepare_alt_odb (); errno =ENOENT ; -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);if (!lstat (*path, st))return 0 ;
@@ -899,7 +898,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path) most_interesting_errno = errno;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);if (fd >=0 )
sha1_name.c +2 -1
@@ -10,6 +10,7 @@#include "dir.h" #include "sha1-array.h" #include "packfile.h" +#include "repository.h" static int get_oid_oneline (const char *,struct object_id *,struct commit_list *);
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)*/ fakeent =alloc_alt_odb (get_object_directory ()); } - fakeent->next = alt_odb_list; + fakeent->next = the_repository->objects.alt_odb_list;for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {int pos;
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
Re: [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store
Quoted 70 lines · Stefan Beller
In a process with multiple repositories open, alternates should be associated to a single repository and not shared globally. Move
alt_odb_listandalt_odb_tailintothe_repositoryand adjust callers to reflect this.Now that the alternative object data base is per repository, we're leaking its memory upon freeing a repository. The next patch plugs this hole.
No functional change intended.
Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>--- builtin/fsck.c | 4 +++- cache.h | 4 ++-- object-store.h | 7 ++++++- packfile.c | 3 ++- sha1_file.c | 25 ++++++++++++------------- sha1_name.c | 3 ++- 6 files changed, 27 insertions(+), 19 deletions(-)builtin/fsck.c +3
-1@@ -1,5 +1,6 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "commit.h" #include "tree.h" @@ -694,7 +695,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)fsck_object_dir (get_object_directory ());prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; + alt; alt = alt->next)fsck_object_dir (alt->path);if (check_full) {cache.h +2
-2@@ -1573,7 +1573,7 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);extern void schedule_dir_for_removal (const char *name,int len);extern void remove_scheduled_dirs (void ); -extern struct alternate_object_database { +struct alternate_object_database {struct alternate_object_database *next;/* see alt_scratch_buf() */ @@ -1591,7 +1591,7 @@ extern struct alternate_object_database {struct oid_array loose_objects_cache;char path[FLEX_ARRAY ]; -} *alt_odb_list; +};extern void prepare_alt_odb (void );extern char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);object-store.h +0
-0
object-store.h line 5
Quoted 5 lines
object-store.h +2
-0@@ -1,14 +1,19 @@#ifndef OBJECT_STORE_H #define OBJECT_STORE_H +#include "cache.h" +
Maybe we would want to move the definition of the alternate store to this header file? That way we don't need to include cache.h here.
Quoted 163 lines
object-store.h +4
-1struct raw_object_store {/* * Path to the repository's object store. * Cannot be NULL after initialization. */ char *objectdir; + +struct alternate_object_database *alt_odb_list; +struct alternate_object_database **alt_odb_tail; }; -#define RAW_OBJECT_STORE_INIT { NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }void raw_object_store_clear (struct raw_object_store *o);packfile.c +2
-1@@ -1,6 +1,7 @@#include "cache.h" #include "mru.h" #include "pack.h" +#include "repository.h" #include "dir.h" #include "mergesort.h" #include "packfile.h" @@ -880,7 +881,7 @@ void prepare_packed_git(void)return ;prepare_packed_git_one (get_object_directory (),1 );prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();prepare_packed_git_mru ();sha1_file.c +12
-13@@ -22,6 +22,7 @@#include "pack-revindex.h" #include "sha1-lookup.h" #include "bulk-checkin.h" +#include "repository.h" #include "streaming.h" #include "dir.h" #include "mru.h" @@ -346,9 +347,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,return buf->buf; } -struct alternate_object_database *alt_odb_list; -static struct alternate_object_database **alt_odb_tail; -/* * Return non-zero iff the path is usable as an alternate object database. */ @@ -368,7 +366,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)* Prevent the common mistake of listing the same * thing twice, or object directory itself. */ -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {if (!fspathcmp (path->buf, alt->path))return 0 ; }@@ -428,8 +426,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base, ent =alloc_alt_odb (pathbuf.buf);/* add the alternate entry */ - *alt_odb_tail = ent; - alt_odb_tail = &(ent->next); + *the_repository->objects.alt_odb_tail = ent; + the_repository->objects.alt_odb_tail = &(ent->next); ent->next =NULL ;/* recursively add alternates */ @@ -563,7 +561,7 @@ void add_to_alternates_file(const char *reference)fprintf_or_die (out,"%s\n" , reference);if (commit_lock_file (&lock))die_errno ("unable to move new alternates file into place" ); -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)link_alt_odb_entries (reference,'\n' ,NULL ,0 ); }free (alts);@@ -661,7 +659,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)int r =0 ;prepare_alt_odb (); -for (ent = alt_odb_list; ent; ent = ent->next) { +for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) { r =fn (ent, cb);if (r)break ;@@ -673,12 +671,13 @@ void prepare_alt_odb(void) {const char *alt; -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)return ; alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - alt_odb_tail = &alt_odb_list; + the_repository->objects.alt_odb_tail = + &the_repository->objects.alt_odb_list;link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 );read_info_alternates (get_object_directory (),0 );@@ -717,7 +716,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen) {struct alternate_object_database *alt;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {const char *path =alt_sha1_path (alt, sha1);if (check_and_freshen_file (path, freshen))return 1 ;@@ -873,7 +872,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,prepare_alt_odb (); errno =ENOENT ; -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);if (!lstat (*path, st))return 0 ;@@ -899,7 +898,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path) most_interesting_errno = errno;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);if (fd >=0 )sha1_name.c +2
-1@@ -10,6 +10,7 @@#include "dir.h" #include "sha1-array.h" #include "packfile.h" +#include "repository.h" static int get_oid_oneline (const char *,struct object_id *,struct commit_list *);@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)*/ fakeent =alloc_alt_odb (get_object_directory ()); } - fakeent->next = alt_odb_list; + fakeent->next = the_repository->objects.alt_odb_list;for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {int pos;Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
Signature · 1 line
Brandon Williams
Re: [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store
Quoted 6 lines · Brandon Williams
Review
Quoted 2 lines
(quoted diff) +2
-0+#include "cache.h" +Maybe we would want to move the definition of the alternate store to this header file? That way we don't need to include
cache.hhere.
Sounds good. I'll take a look at that.
Stefan
PATCH 3/26 object-store: free alt_odb_list
In reply to Stefan BellerFree the memory and reset alt_odb_{list, tail} to NULL.
--- object.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
object.c +17 -0
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags) } } +static void free_alt_odb (struct alternate_object_database *alt) +{ +strbuf_release (&alt->scratch); +oid_array_clear (&alt->loose_objects_cache); +} + +static void free_alt_odbs (struct raw_object_store *o) +{ +while (o->alt_odb_list) { +free_alt_odb (o->alt_odb_list); + o->alt_odb_list = o->alt_odb_list->next; + } +} +void raw_object_store_clear (struct raw_object_store *o) {free (o->objectdir); + +free_alt_odbs (o); + o->alt_odb_tail =NULL ; }
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 4/26 object-store: move packed_git and packed_git_mru to object store
In reply to Stefan BellerIn a process with multiple repositories open, packfile accessors should be associated to a single repository and not shared globally. Move packed_git and packed_git_mru into the_repository and adjust callers to reflect this.
Patch generated by
1. Moving the struct packed_git declaration to object-store.h and packed_git, packed_git_mru globals to struct object_store.
2. Applying the semantic patch contrib/coccinelle/refactoring/packed_git.cocci to adjust callers. This semantic patch is placed in a sub directory of the coccinelle contrib dir, as this semantic patch is not expected to be of general usefulness; it is only useful during developing this series and merging it with other topics in flight. At a later date, just delete that semantic patch.
3. Applying line wrapping fixes from "make style" to break the resulting long lines.
4. Adding missing #includes of repository.h and object-store.h where needed.
5. As the packfiles are now owned by an objectstore/repository, which is ephemeral unlike globals, we introduce memory leaks. So address them in raw_object_store_clear().
--- builtin/count-objects.c | 6 ++-- builtin/fsck.c | 7 +++-- builtin/gc.c | 4 ++- builtin/index-pack.c | 1 + builtin/pack-objects.c | 21 ++++++++----- builtin/pack-redundant.c | 6 ++-- builtin/receive-pack.c | 1 + cache.h | 28 ------------------ contrib/coccinelle/refactoring/packed_git.cocci | 7 +++++ fast-import.c | 6 ++-- http-backend.c | 6 ++-- http-push.c | 1 + http-walker.c | 1 + http.c | 1 + mru.h | 1 + object-store.h | 32 +++++++++++++++++++- object.c | 6 ++++ pack-bitmap.c | 4 ++- pack-check.c | 1 + pack-revindex.c | 1 + packfile.c | 39 +++++++++++++------------ reachable.c | 1 + server-info.c | 6 ++-- sha1_name.c | 5 ++-- 24 files changed, 120 insertions(+), 72 deletions(-) create mode 100644 contrib/coccinelle/refactoring/packed_git.cocci
builtin/count-objects.c +4 -2
@@ -7,6 +7,8 @@#include "cache.h" #include "config.h" #include "dir.h" +#include "repository.h" +#include "object-store.h" #include "builtin.h" #include "parse-options.h" #include "quote.h"
@@ -120,9 +122,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)struct strbuf loose_buf =STRBUF_INIT ;struct strbuf pack_buf =STRBUF_INIT ;struct strbuf garbage_buf =STRBUF_INIT ; -if (!packed_git) +if (!the_repository->objects.packed_git)prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ;if (open_pack_index (p))
builtin/fsck.c +5 -2
@@ -2,6 +2,7 @@#include "cache.h" #include "repository.h" #include "config.h" +#include "object-store.h" #include "commit.h" #include "tree.h" #include "blob.h"
@@ -707,7 +708,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)prepare_packed_git ();if (show_progress) { -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {if (open_pack_index (p))continue ; total += p->num_objects;
@@ -715,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) progress =start_progress (_ ("Checking objects" ), total); } -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {/* verify gives error messages itself */ if (verify_pack (p, fsck_obj_buffer, progress, count))
builtin/gc.c +3 -1
@@ -11,6 +11,7 @@*/ #include "builtin.h" +#include "repository.h" #include "config.h" #include "tempfile.h" #include "lockfile.h"
@@ -18,6 +19,7 @@#include "run-command.h" #include "sigchain.h" #include "argv-array.h" +#include "object-store.h" #include "commit.h" #include "packfile.h"
@@ -173,7 +175,7 @@ static int too_many_packs(void)return 0 ;prepare_packed_git (); -for (cnt =0 , p = packed_git; p; p = p->next) { +for (cnt =0 , p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ;if (p->pack_keep)
builtin/index-pack.c +1 -0
@@ -12,6 +12,7 @@#include "exec_cmd.h" #include "streaming.h" #include "thread-utils.h" +#include "object-store.h" #include "packfile.h" static const char index_pack_usage[] =
builtin/pack-objects.c +13 -8
@@ -1,6 +1,8 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" +#include "object-store.h" #include "attr.h" #include "object.h" #include "blob.h"
@@ -1023,7 +1025,8 @@ static int want_object_in_pack(const struct object_id *oid,return want; } -for (entry = packed_git_mru.head; entry; entry = entry->next) { +for (entry = the_repository->objects.packed_git_mru.head; entry; + entry = entry->next) {struct packed_git *p = entry->item;off_t offset;
@@ -1041,7 +1044,8 @@ static int want_object_in_pack(const struct object_id *oid, } want =want_found_object (exclude, p);if (!exclude && want >0 ) -mru_mark (&packed_git_mru, entry); +mru_mark (&the_repository->objects.packed_git_mru, + entry);if (want !=-1 )return want; }
@@ -2648,7 +2652,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)memset (&in_pack,0 ,sizeof (in_pack)); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {struct object_id oid;struct object *o;
@@ -2711,7 +2715,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)static struct packed_git *last_found = (void *)1 ;struct packed_git *p; - p = (last_found != (void *)1 ) ? last_found : packed_git; + p = (last_found != (void *)1 ) ? last_found : + the_repository->objects.packed_git;while (p) { if ((!p->pack_local || p->pack_keep) &&
@@ -2720,7 +2725,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)return 1 ; }if (p == last_found) - p = packed_git; + p = the_repository->objects.packed_git;else p = p->next;if (p == last_found)
@@ -2756,7 +2761,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)uint32_t i;struct object_id oid; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local || p->pack_keep)continue ;
@@ -3119,7 +3124,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)prepare_packed_git ();if (ignore_packed_keep) {struct packed_git *p; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)if (p->pack_local && p->pack_keep)break ;if (!p)/* no keep-able packs found */
@@ -3132,7 +3137,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)* also covers non-local objects */ struct packed_git *p; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local) { have_non_local_packs =1 ;break ;
builtin/pack-redundant.c +4 -2
@@ -7,6 +7,8 @@*/ #include "builtin.h" +#include "repository.h" +#include "object-store.h" #include "packfile.h" #define BLKSIZE 512
@@ -571,7 +573,7 @@ static struct pack_list * add_pack(struct packed_git *p)static struct pack_list *add_pack_file (const char *filename) { -struct packed_git *p = packed_git; +struct packed_git *p = the_repository->objects.packed_git;if (strlen (filename) <40 )die ("Bad pack filename: %s" , filename);
@@ -586,7 +588,7 @@ static struct pack_list * add_pack_file(const char *filename)static void load_all (void ) { -struct packed_git *p = packed_git; +struct packed_git *p = the_repository->objects.packed_git;while (p) {add_pack (p);
builtin/receive-pack.c +1 -0
@@ -7,6 +7,7 @@#include "sideband.h" #include "run-command.h" #include "exec_cmd.h" +#include "object-store.h" #include "commit.h" #include "object.h" #include "remote.h"
cache.h +0 -28
@@ -1633,34 +1633,6 @@ struct pack_window {unsigned int inuse_cnt; }; -extern struct packed_git { -struct packed_git *next; -struct pack_window *windows; -off_t pack_size; -const void *index_data; -size_t index_size; -uint32_t num_objects; -uint32_t num_bad_objects; -unsigned char *bad_object_sha1; -int index_version; -time_t mtime; -int pack_fd; -unsigned pack_local:1 , - pack_keep:1 , - freshened:1 , - do_not_close:1 ; -unsigned char sha1[20 ]; -struct revindex_entry *revindex; -/* something like ".git/objects/pack/xxxxx.pack" */ -char pack_name[FLEX_ARRAY ];/* more */ -} *packed_git; - -/* -* A most-recently-used ordered version of the packed_git list, which can -* be iterated instead of packed_git (and marked via mru_mark). -*/ -extern struct mru packed_git_mru; -struct pack_entry {off_t offset;unsigned char sha1[20 ];
contrib/coccinelle/refactoring/packed_git.cocci new file +7 -0
@@ -0,0 +1,7 @@ +@@ @@ +- packed_git ++ the_repository->objects.packed_git + +@@ @@ +- packed_git_mru ++ the_repository->objects.packed_git_mru
fast-import.c +4 -2
@@ -154,8 +154,10 @@ Format of STDIN stream:#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "lockfile.h" +#include "object-store.h" #include "object.h" #include "blob.h" #include "tree.h"
@@ -1110,7 +1112,7 @@ static int store_object(if (e->idx.offset) { duplicate_count_by_type[type]++;return 1 ; - }else if (find_sha1_pack (oid.hash, packed_git)) { + }else if (find_sha1_pack (oid.hash, the_repository->objects.packed_git)) { e->type = type; e->pack_id =MAX_PACK_ID ; e->idx.offset =1 ;/* just not zero! */
@@ -1305,7 +1307,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark) duplicate_count_by_type[OBJ_BLOB ]++;truncate_pack (&checkpoint); - }else if (find_sha1_pack (oid.hash, packed_git)) { + }else if (find_sha1_pack (oid.hash, the_repository->objects.packed_git)) { e->type =OBJ_BLOB ; e->pack_id =MAX_PACK_ID ; e->idx.offset =1 ;/* just not zero! */
http-backend.c +4 -2
@@ -1,7 +1,9 @@#include "cache.h" #include "config.h" +#include "repository.h" #include "refs.h" #include "pkt-line.h" +#include "object-store.h" #include "object.h" #include "tag.h" #include "exec_cmd.h"
@@ -518,13 +520,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)select_getanyfile (hdr);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_local) cnt++; }strbuf_grow (&buf, cnt *53 +2 ); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_local)strbuf_addf (&buf,"P %s\n" , p->pack_name + objdirlen +6 ); }
http-push.c +1 -0
@@ -1,4 +1,5 @@#include "cache.h" +#include "object-store.h" #include "commit.h" #include "tag.h" #include "blob.h"
http-walker.c +1 -0
@@ -4,6 +4,7 @@#include "http.h" #include "list.h" #include "transport.h" +#include "object-store.h" #include "packfile.h" struct alt_base {
http.c +1 -0
@@ -1,6 +1,7 @@#include "git-compat-util.h" #include "http.h" #include "config.h" +#include "object-store.h" #include "pack.h" #include "sideband.h" #include "run-command.h"
mru.h +1 -0
@@ -37,6 +37,7 @@ struct mru_entry {struct mru {struct mru_entry *head, *tail; }; +#define MRU_INIT { NULL, NULL }void mru_append (struct mru *mru,void *item);void mru_mark (struct mru *mru,struct mru_entry *entry);
object-store.h +31 -1
@@ -2,6 +2,7 @@#define OBJECT_STORE_H #include "cache.h" +#include "mru.h" struct raw_object_store {/*
@@ -10,11 +11,40 @@ struct raw_object_store {*/ char *objectdir; +struct packed_git *packed_git; +/* +* A most-recently-used ordered version of the packed_git list, which can +* be iterated instead of packed_git (and marked via mru_mark). +*/ +struct mru packed_git_mru; +struct alternate_object_database *alt_odb_list; struct alternate_object_database **alt_odb_tail; }; -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL }void raw_object_store_clear (struct raw_object_store *o); +struct packed_git { +struct packed_git *next; +struct pack_window *windows; +off_t pack_size; +const void *index_data; +size_t index_size; +uint32_t num_objects; +uint32_t num_bad_objects; +unsigned char *bad_object_sha1; +int index_version; +time_t mtime; +int pack_fd; +unsigned pack_local:1 , + pack_keep:1 , + freshened:1 , + do_not_close:1 ; +unsigned char sha1[20 ]; +struct revindex_entry *revindex; +/* something like ".git/objects/pack/xxxxx.pack" */ +char pack_name[FLEX_ARRAY ];/* more */ +}; +#endif /* OBJECT_STORE_H */
object.c +6 -0
@@ -466,4 +466,10 @@ void raw_object_store_clear(struct raw_object_store *o)free_alt_odbs (o); o->alt_odb_tail =NULL ; + +mru_clear (&o->packed_git_mru); +/* +* TODO: call close_all_packs once migrated to +* take an object store argument +*/ }
pack-bitmap.c +3 -1
@@ -5,11 +5,13 @@#include "revision.h" #include "progress.h" #include "list-objects.h" +#include "object-store.h" #include "pack.h" #include "pack-bitmap.h" #include "pack-revindex.h" #include "pack-objects.h" #include "packfile.h" +#include "repository.h" /* * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +337,7 @@ static int open_pack_bitmap(void)assert (!bitmap_git.map && !bitmap_git.loaded);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (open_pack_bitmap_1 (p) ==0 ) ret =0 ; }
pack-check.c +1 -0
@@ -1,5 +1,6 @@#include "cache.h" #include "pack.h" +#include "object-store.h" #include "pack-revindex.h" #include "progress.h" #include "packfile.h"
pack-revindex.c +1 -0
@@ -1,5 +1,6 @@#include "cache.h" #include "pack-revindex.h" +#include "object-store.h" /* * Pack index for existing packs give us easy access to the offsets into
packfile.c +20 -19
@@ -7,6 +7,7 @@#include "packfile.h" #include "delta.h" #include "list.h" +#include "object-store.h" #include "streaming.h" #include "sha1-lookup.h"
@@ -40,8 +41,6 @@ static unsigned int pack_open_fds;static unsigned int pack_max_fds;static size_t peak_pack_mapped;static size_t pack_mapped; -struct packed_git *packed_git; -struct mru packed_git_mru;#define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; }
@@ -241,7 +240,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -311,7 +310,7 @@ void close_all_packs(void) {struct packed_git *p; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)if (p->do_not_close)die ("BUG: want to close pack marked 'do-not-close'" );else
@@ -379,7 +378,7 @@ static int close_one_pack(void)struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -677,8 +676,8 @@ void install_packed_git(struct packed_git *pack)if (pack->pack_fd !=-1 ) pack_open_fds++; - pack->next = packed_git; - packed_git = pack; + pack->next = the_repository->objects.packed_git; + the_repository->objects.packed_git = pack; }void (*report_garbage)(unsigned seen_bits,const char *path);
@@ -760,7 +759,8 @@ static void prepare_packed_git_one(char *objdir, int local) base_len = path.len;if (strip_suffix_mem (path.buf, &base_len,".idx" )) {/* Don't reopen a pack we already have. */ -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {size_t len; if (strip_suffix(p->pack_name, ".pack", &len) && len == base_len &&
@@ -810,7 +810,7 @@ unsigned long approximate_object_count(void)prepare_packed_git (); count =0 ; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (open_pack_index (p))continue ; count += p->num_objects;
@@ -859,17 +859,18 @@ static int sort_pack(const void *a_, const void *b_)static void rearrange_packed_git (void ) { - packed_git =llist_mergesort (packed_git, get_next_packed_git, - set_next_packed_git, sort_pack); + the_repository->objects.packed_git =llist_mergesort ( + the_repository->objects.packed_git, get_next_packed_git, + set_next_packed_git, sort_pack); }static void prepare_packed_git_mru (void ) {struct packed_git *p; -mru_clear (&packed_git_mru); -for (p = packed_git; p; p = p->next) -mru_append (&packed_git_mru, p); +mru_clear (&the_repository->objects.packed_git_mru); +for (p = the_repository->objects.packed_git; p; p = p->next) +mru_append (&the_repository->objects.packed_git_mru, p); }static int prepare_packed_git_run_once =0 ;
@@ -1003,7 +1004,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)struct packed_git *p;unsigned i; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)for (i =0 ; i < p->num_bad_objects; i++)if (!hashcmp (sha1, p->bad_object_sha1 +20 * i))return p;
@@ -1835,12 +1836,12 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)struct mru_entry *p;prepare_packed_git (); -if (!packed_git) +if (!the_repository->objects.packed_git)return 0 ; -for (p = packed_git_mru.head; p; p = p->next) { +for (p = the_repository->objects.packed_git_mru.head; p; p = p->next) {if (fill_pack_entry (sha1, e, p->item)) { -mru_mark (&packed_git_mru, p); +mru_mark (&the_repository->objects.packed_git_mru, p);return 1 ; } }
@@ -1887,7 +1888,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)int pack_errors =0 ;prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ;if (open_pack_index (p)) {
reachable.c +1 -0
@@ -1,5 +1,6 @@#include "cache.h" #include "refs.h" +#include "object-store.h" #include "tag.h" #include "commit.h" #include "blob.h"
server-info.c +4 -2
@@ -1,5 +1,7 @@#include "cache.h" +#include "repository.h" #include "refs.h" +#include "object-store.h" #include "object.h" #include "commit.h" #include "tag.h"
@@ -200,7 +202,7 @@ static void init_pack_info(const char *infofile, int force) objdirlen =strlen (objdir);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {/* we ignore things on alternate path since they are * not available to the pullers in general. */
@@ -210,7 +212,7 @@ static void init_pack_info(const char *infofile, int force) } num_pack = i; info =xcalloc (num_pack,sizeof (struct pack_info *)); -for (i =0 , p = packed_git; p; p = p->next) { +for (i =0 , p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ; info[i] =xcalloc (1 ,sizeof (struct pack_info ));
sha1_name.c +3 -2
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)struct packed_git *p;prepare_packed_git (); -for (p = packed_git; p && !ds->ambiguous; p = p->next) +for (p = the_repository->objects.packed_git; p && !ds->ambiguous; + p = p->next)unique_in_pack (p, ds); }
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)struct packed_git *p;prepare_packed_git (); -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)find_abbrev_len_for_pack (p, mad); }
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 5/26 object-store: close all packs upon clearing the object store
In reply to Stefan Beller--- builtin/am.c | 2 +- builtin/clone.c | 2 +- builtin/fetch.c | 2 +- builtin/merge.c | 2 +- builtin/receive-pack.c | 2 +- object.c | 6 ++---- packfile.c | 4 ++-- packfile.h | 2 +- 8 files changed, 10 insertions(+), 12 deletions(-)
builtin/am.c +1 -1
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)*/ if (!state->rebasing) {am_destroy (state); -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }
builtin/clone.c +1 -1
@@ -1200,7 +1200,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)transport_disconnect (transport);if (option_dissociate) { -close_all_packs (); +close_all_packs (&the_repository->objects);dissociate_from_references (); }
builtin/fetch.c +1 -1
@@ -1407,7 +1407,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)string_list_clear (&list,0 ); -close_all_packs (); +close_all_packs (&the_repository->objects);argv_array_pushl (&argv_gc_auto,"gc" ,"--auto" ,NULL );if (verbosity <0 )
builtin/merge.c +1 -1
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,* We ignore errors in 'gc --auto', since the * user should see them. */ -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }
builtin/receive-pack.c +1 -1
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix) proc.git_cmd =1 ; proc.argv = argv_gc_auto; -close_all_packs (); +close_all_packs (&the_repository->objects);if (!start_command (&proc)) {if (use_sideband)copy_to_sideband (proc.err,-1 ,NULL );
object.c +2 -4
@@ -4,6 +4,7 @@#include "tree.h" #include "commit.h" #include "tag.h" +#include "packfile.h" static struct object **obj_hash;static int nr_objs, obj_hash_size;
@@ -468,8 +469,5 @@ void raw_object_store_clear(struct raw_object_store *o) o->alt_odb_tail =NULL ;mru_clear (&o->packed_git_mru); -/* -* TODO: call close_all_packs once migrated to -* take an object store argument -*/ +close_all_packs (o); }
packfile.c +2 -2
@@ -306,11 +306,11 @@ static void close_pack(struct packed_git *p)close_pack_index (p); } -void close_all_packs (void ) +void close_all_packs (struct raw_object_store *o) {struct packed_git *p; -for (p = the_repository->objects.packed_git; p; p = p->next) +for (p = o->packed_git; p; p = p->next)if (p->do_not_close)die ("BUG: want to close pack marked 'do-not-close'" );else
packfile.h +1 -1
@@ -61,7 +61,7 @@ extern void close_pack_index(struct packed_git *);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void close_pack_windows (struct packed_git *); -extern void close_all_packs (void ); +extern void close_all_packs (struct raw_object_store *o);extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 6/26 pack: move prepare_packed_git_run_once to object store
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
Each repository's object store can be initialized independently, so they must not share a run_once variable.
--- object-store.h | 8 +++++++- packfile.c | 7 +++---- 2 files changed, 10 insertions(+), 5 deletions(-)
object-store.h +7 -1
@@ -20,8 +20,14 @@ struct raw_object_store {struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; + +/* +* Whether packed_git has already been populated with this repository's +* packs. +*/ + unsigned packed_git_initialized : 1; }; -#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0 }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +3 -4
@@ -873,12 +873,11 @@ static void prepare_packed_git_mru(void)mru_append (&the_repository->objects.packed_git_mru, p); } -static int prepare_packed_git_run_once =0 ;void prepare_packed_git (void ) {struct alternate_object_database *alt; -if (prepare_packed_git_run_once) +if (the_repository->objects.packed_git_initialized)return ;prepare_packed_git_one (get_object_directory (),1 );prepare_alt_odb ();
@@ -886,13 +885,13 @@ void prepare_packed_git(void)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();prepare_packed_git_mru (); - prepare_packed_git_run_once =1 ; + the_repository->objects.packed_git_initialized =1 ; }void reprepare_packed_git (void ) { approximate_object_count_valid =0 ; - prepare_packed_git_run_once =0 ; + the_repository->objects.packed_git_initialized =0 ;prepare_packed_git (); }
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 7/26 pack: move approximate object count to object store
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
The approximate_object_count() function maintains a rough count of objects in a repository to estimate how long object name abbreviates should be. Object names are scoped to a repository and the appropriate length may differ by repository, so the object count should not be global.
--- object-store.h | 10 +++++++++- packfile.c | 11 +++++------ 2 files changed, 14 insertions(+), 7 deletions(-)
object-store.h +9 -1
@@ -21,13 +21,21 @@ struct raw_object_store {struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; +/* +* A fast, rough count of the number of objects in the repository. +* These two fields are not meant for direct access. Use +* approximate_object_count() instead. +*/ +unsigned long approximate_object_count; + unsigned approximate_object_count_valid : 1; +/* * Whether packed_git has already been populated with this repository's * packs. */ unsigned packed_git_initialized : 1; }; -#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0 } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +5 -6
@@ -793,8 +793,6 @@ static void prepare_packed_git_one(char *objdir, int local)strbuf_release (&path); } -static int approximate_object_count_valid; -/* * Give a fast, rough count of the number of objects in the repository. This * ignores loose objects completely. If you have a lot of them, then either
@@ -804,8 +802,8 @@ static int approximate_object_count_valid;*/ unsigned long approximate_object_count (void ) { -static unsigned long count; -if (!approximate_object_count_valid) { +if (!the_repository->objects.approximate_object_count_valid) { +unsigned long count;struct packed_git *p;prepare_packed_git ();
@@ -815,8 +813,9 @@ unsigned long approximate_object_count(void)continue ; count += p->num_objects; } + the_repository->objects.approximate_object_count = count; } -return count; +return the_repository->objects.approximate_object_count; }static void *get_next_packed_git (const void *p)
@@ -890,7 +889,7 @@ void prepare_packed_git(void)void reprepare_packed_git (void ) { - approximate_object_count_valid =0 ; + the_repository->objects.approximate_object_count_valid =0 ; the_repository->objects.packed_git_initialized =0 ;prepare_packed_git (); }
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 8/26 sha1_file: add raw_object_store argument to alt_odb_usable
In reply to Stefan BellerAdd a raw_object_store to alt_odb_usable to be more specific about which repository to act on. The choice of the repository is delegated to its only caller link_alt_odb_entry.
--- sha1_file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
sha1_file.c +5 -3
@@ -350,7 +350,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,/* * Return non-zero iff the path is usable as an alternate object database. */ -static int alt_odb_usable (struct strbuf *path,const char *normalized_objdir) +static int alt_odb_usable (struct raw_object_store *o, +struct strbuf *path, +const char *normalized_objdir) {struct alternate_object_database *alt;
@@ -366,7 +368,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)* Prevent the common mistake of listing the same * thing twice, or object directory itself. */ -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +for (alt = o->alt_odb_list; alt; alt = alt->next) {if (!fspathcmp (path->buf, alt->path))return 0 ; }
@@ -418,7 +420,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,while (pathbuf.len && pathbuf.buf[pathbuf.len -1 ] =='/' )strbuf_setlen (&pathbuf, pathbuf.len -1 ); -if (!alt_odb_usable (&pathbuf, normalized_objdir)) { +if (!alt_odb_usable (&the_repository->objects, &pathbuf, normalized_objdir)) {strbuf_release (&pathbuf);return -1 ; }
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 9/26 sha1_file: add repository argument to link_alt_odb_entry
In reply to Stefan BellerAdd a repository argument to allow the link_alt_odb_entry caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
Since the implementation does not yet work with other repositories, use a wrapper macro to enforce that the caller passes in the_repository as the first argument. It would be more appealing to use BUILD_ASSERT_OR_ZERO to enforce this, but that doesn't work because it requires a compile-time constant and common compilers like gcc 4.8.4 do not consider "r == the_repository" a compile-time constant.
This and the following three patches add repository arguments to link_alt_odb_entry, read_info_alternates, link_alt_odb_entries and prepare_alt_odb. Three out of the four functions are found in a recursive call chain, calling each other, and one of them accesses the repositories objectdir (which was migrated; it was an obvious choice) and ignore_env (which we need to keep in the repository struct for clarify); hence we will pass through the repository unlike just the object store object + the ignore_env flag.
--- sha1_file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
sha1_file.c +5 -3
@@ -394,8 +394,9 @@ static int alt_odb_usable(struct raw_object_store *o,* terminating NUL. */ static void read_info_alternates (const char * relative_base,int depth); -static int link_alt_odb_entry (const char *entry,const char *relative_base, -int depth,const char *normalized_objdir) +#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n) +static int link_alt_odb_entry_the_repository (const char *entry, +const char *relative_base,int depth,const char *normalized_objdir) {struct alternate_object_database *ent;struct strbuf pathbuf =STRBUF_INIT ;
@@ -492,7 +493,8 @@ static void link_alt_odb_entries(const char *alt, int sep, alt =parse_alt_odb_entry (alt, sep, &entry);if (!entry.len)continue ; -link_alt_odb_entry (entry.buf, relative_base, depth, objdirbuf.buf); +link_alt_odb_entry (the_repository, entry.buf, + relative_base, depth, objdirbuf.buf); }strbuf_release (&entry);strbuf_release (&objdirbuf);
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 10/26 sha1_file: add repository argument to read_info_alternates
In reply to Stefan BellerSee previous patch for explanation.
--- sha1_file.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
sha1_file.c +7 -4
@@ -393,7 +393,9 @@ static int alt_odb_usable(struct raw_object_store *o,* SHA1, an extra slash for the first level indirection, and the * terminating NUL. */ -static void read_info_alternates (const char * relative_base,int depth); +#define read_info_alternates (r, rb, d) read_info_alternates_##r(rb, d) +static void read_info_alternates_the_repository (const char *relative_base, +int depth);#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)static int link_alt_odb_entry_the_repository (const char *entry,const char *relative_base,int depth,const char *normalized_objdir)
@@ -434,7 +436,7 @@ static int link_alt_odb_entry_the_repository(const char *entry, ent->next =NULL ;/* recursively add alternates */ -read_info_alternates (pathbuf.buf, depth +1 ); +read_info_alternates (the_repository, pathbuf.buf, depth +1 );strbuf_release (&pathbuf);return 0 ;
@@ -500,7 +502,8 @@ static void link_alt_odb_entries(const char *alt, int sep,strbuf_release (&objdirbuf); } -static void read_info_alternates (const char * relative_base,int depth) +static void read_info_alternates_the_repository (const char *relative_base, +int depth) {char *path;struct strbuf buf =STRBUF_INIT ;
@@ -684,7 +687,7 @@ void prepare_alt_odb(void) &the_repository->objects.alt_odb_list;link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 ); -read_info_alternates (get_object_directory (),0 ); +read_info_alternates (the_repository,get_object_directory (),0 ); }/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 11/26 sha1_file: add repository argument to link_alt_odb_entries
In reply to Stefan BellerSee previous patch for explanation.
--- sha1_file.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
sha1_file.c +13 -6
@@ -471,8 +471,12 @@ static const char *parse_alt_odb_entry(const char *string,return end; } -static void link_alt_odb_entries (const char *alt,int sep, -const char *relative_base,int depth) +#define link_alt_odb_entries (r, a, s, rb, d) \ + link_alt_odb_entries_##r(a, s, rb, d) +static void link_alt_odb_entries_the_repository (const char *alt, +int sep, +const char *relative_base, +int depth) {struct strbuf objdirbuf =STRBUF_INIT ;struct strbuf entry =STRBUF_INIT ;
@@ -515,7 +519,7 @@ static void read_info_alternates_the_repository(const char *relative_base,return ; } -link_alt_odb_entries (buf.buf,'\n' , relative_base, depth); +link_alt_odb_entries (the_repository, buf.buf,'\n' , relative_base, depth);strbuf_release (&buf);free (path); }
@@ -569,7 +573,8 @@ void add_to_alternates_file(const char *reference)if (commit_lock_file (&lock))die_errno ("unable to move new alternates file into place" );if (the_repository->objects.alt_odb_tail) -link_alt_odb_entries (reference,'\n' ,NULL ,0 ); +link_alt_odb_entries (the_repository, reference, +'\n' ,NULL ,0 ); }free (alts); }
@@ -582,7 +587,8 @@ void add_to_alternates_memory(const char *reference)*/ prepare_alt_odb (); -link_alt_odb_entries (reference,'\n' ,NULL ,0 ); +link_alt_odb_entries (the_repository, reference, +'\n' ,NULL ,0 ); }/*
@@ -685,7 +691,8 @@ void prepare_alt_odb(void) the_repository->objects.alt_odb_tail = &the_repository->objects.alt_odb_list; -link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (the_repository, alt, +PATH_SEP ,NULL ,0 );read_info_alternates (the_repository,get_object_directory (),0 ); }
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 12/26 sha1_file: add repository argument to prepare_alt_odb
In reply to Stefan BellerSee previous patch for explanation.
While at it, move the declaration to object-store.h, where it should be easier to find.
--- builtin/fsck.c | 2 +- cache.h | 1 - object-store.h | 3 +++ packfile.c | 2 +- sha1_file.c | 13 +++++++------ sha1_name.c | 3 ++- 6 files changed, 14 insertions(+), 10 deletions(-)
builtin/fsck.c +1 -1
@@ -695,7 +695,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) }else {fsck_object_dir (get_object_directory ()); -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)fsck_object_dir (alt->path);
cache.h +0 -1
@@ -1592,7 +1592,6 @@ struct alternate_object_database {char path[FLEX_ARRAY ]; }; -extern void prepare_alt_odb (void );extern char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);extern int foreach_alt_odb (alt_odb_fn ,void *);
object-store.h +3 -0
@@ -61,4 +61,7 @@ struct packed_git { char pack_name[FLEX_ARRAY]; /* more */ }; +#define prepare_alt_odb (r) prepare_alt_odb_##r() +extern void prepare_alt_odb_the_repository (void ); +#endif /* OBJECT_STORE_H */
packfile.c +1 -1
@@ -879,7 +879,7 @@ void prepare_packed_git(void)if (the_repository->objects.packed_git_initialized)return ;prepare_packed_git_one (get_object_directory (),1 ); -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();
sha1_file.c +7 -6
@@ -23,6 +23,7 @@#include "sha1-lookup.h" #include "bulk-checkin.h" #include "repository.h" +#include "object-store.h" #include "streaming.h" #include "dir.h" #include "mru.h"
@@ -585,7 +586,7 @@ void add_to_alternates_memory(const char *reference)* Make sure alternates are initialized, or else our entry may be * overwritten when they are. */ -prepare_alt_odb (); +prepare_alt_odb (the_repository);link_alt_odb_entries (the_repository, reference,'\n' ,NULL ,0 );
@@ -671,7 +672,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)struct alternate_object_database *ent;int r =0 ; -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) { r =fn (ent, cb);if (r)
@@ -680,7 +681,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb (void ) +void prepare_alt_odb_the_repository (void ) {const char *alt;
@@ -729,7 +730,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)static int check_and_freshen_nonlocal (const unsigned char *sha1,int freshen) {struct alternate_object_database *alt; -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {const char *path =alt_sha1_path (alt, sha1);if (check_and_freshen_file (path, freshen))
@@ -884,7 +885,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,if (!lstat (*path, st))return 0 ; -prepare_alt_odb (); +prepare_alt_odb (the_repository); errno =ENOENT ;for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);
@@ -911,7 +912,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)return fd; most_interesting_errno = errno; -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);
sha1_name.c +2 -1
@@ -11,6 +11,7 @@#include "sha1-array.h" #include "packfile.h" #include "repository.h" +#include "object-store.h" static int get_oid_oneline (const char *,struct object_id *,struct commit_list *);
@@ -353,7 +354,7 @@ static int init_object_disambiguation(const char *name, int len, ds->len = len; ds->hex_pfx[len] ='\0' ; -prepare_alt_odb (); +prepare_alt_odb (the_repository);return 0 ; }
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 13/26 sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
In reply to Stefan BellerActually this also allows read_info_alternates and link_alt_odb_entry to handle arbitrary repositories, but link_alt_odb_entries is the most interesting function in this set of functions, hence the commit subject.
These functions span a strongly connected component in the function graph, i.e. the recursive call chain might look like
-> link_alt_odb_entries
-> link_alt_odb_entry
-> read_info_alternates
-> link_alt_odb_entriesThat is why we need to convert all these functions at the same time.
--- cache.h | 4 ++++ sha1_file.c | 36 ++++++++++++++++-------------------- 2 files changed, 20 insertions(+), 20 deletions(-)
cache.h +4 -0
@@ -1590,6 +1590,10 @@ struct alternate_object_database {char loose_objects_subdir_seen[256 ];struct oid_array loose_objects_cache; +/* +* Path to the alternative object store. If this is a relative path, +* it is relative to the current working directory. +*/ char path[FLEX_ARRAY ]; };extern char *compute_alternate_path (const char *path,struct strbuf *err);
sha1_file.c +16 -20
@@ -394,11 +394,10 @@ static int alt_odb_usable(struct raw_object_store *o,* SHA1, an extra slash for the first level indirection, and the * terminating NUL. */ -#define read_info_alternates (r, rb, d) read_info_alternates_##r(rb, d) -static void read_info_alternates_the_repository (const char *relative_base, -int depth); -#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n) -static int link_alt_odb_entry_the_repository (const char *entry, +static void read_info_alternates (struct repository *r, +const char *relative_base, +int depth); +static int link_alt_odb_entry (struct repository *r,const char *entry,const char *relative_base,int depth,const char *normalized_objdir) {struct alternate_object_database *ent;
@@ -424,7 +423,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,while (pathbuf.len && pathbuf.buf[pathbuf.len -1 ] =='/' )strbuf_setlen (&pathbuf, pathbuf.len -1 ); -if (!alt_odb_usable (&the_repository->objects, &pathbuf, normalized_objdir)) { +if (!alt_odb_usable (&r->objects, &pathbuf, normalized_objdir)) {strbuf_release (&pathbuf);return -1 ; }
@@ -432,12 +431,12 @@ static int link_alt_odb_entry_the_repository(const char *entry, ent =alloc_alt_odb (pathbuf.buf);/* add the alternate entry */ - *the_repository->objects.alt_odb_tail = ent; - the_repository->objects.alt_odb_tail = &(ent->next); + *r->objects.alt_odb_tail = ent; + r->objects.alt_odb_tail = &(ent->next); ent->next =NULL ;/* recursively add alternates */ -read_info_alternates (the_repository, pathbuf.buf, depth +1 ); +read_info_alternates (r, pathbuf.buf, depth +1 );strbuf_release (&pathbuf);return 0 ;
@@ -472,12 +471,8 @@ static const char *parse_alt_odb_entry(const char *string,return end; } -#define link_alt_odb_entries (r, a, s, rb, d) \ - link_alt_odb_entries_##r(a, s, rb, d) -static void link_alt_odb_entries_the_repository (const char *alt, -int sep, -const char *relative_base, -int depth) +static void link_alt_odb_entries (struct repository *r,const char *alt, +int sep,const char *relative_base,int depth) {struct strbuf objdirbuf =STRBUF_INIT ;struct strbuf entry =STRBUF_INIT ;
@@ -491,7 +486,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,return ; } -strbuf_add_absolute_path (&objdirbuf,get_object_directory ()); +strbuf_add_absolute_path (&objdirbuf, r->objects.objectdir);if (strbuf_normalize_path (&objdirbuf) <0 )die ("unable to normalize object directory: %s" , objdirbuf.buf);
@@ -500,15 +495,16 @@ static void link_alt_odb_entries_the_repository(const char *alt, alt =parse_alt_odb_entry (alt, sep, &entry);if (!entry.len)continue ; -link_alt_odb_entry (the_repository, entry.buf, +link_alt_odb_entry (r, entry.buf, relative_base, depth, objdirbuf.buf); }strbuf_release (&entry);strbuf_release (&objdirbuf); } -static void read_info_alternates_the_repository (const char *relative_base, -int depth) +static void read_info_alternates (struct repository *r, +const char *relative_base, +int depth) {char *path;struct strbuf buf =STRBUF_INIT ;
@@ -520,7 +516,7 @@ static void read_info_alternates_the_repository(const char *relative_base,return ; } -link_alt_odb_entries (the_repository, buf.buf,'\n' , relative_base, depth); +link_alt_odb_entries (r, buf.buf,'\n' , relative_base, depth);strbuf_release (&buf);free (path); }
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 14/26 sha1_file: allow prepare_alt_odb to handle arbitrary repositories
In reply to Stefan Beller--- object-store.h | 3 +-- sha1_file.c | 21 +++++++++++---------- 2 files changed, 12 insertions(+), 12 deletions(-)
object-store.h +1 -2
@@ -61,7 +61,6 @@ struct packed_git { char pack_name[FLEX_ARRAY]; /* more */ }; -#define prepare_alt_odb (r) prepare_alt_odb_##r() -extern void prepare_alt_odb_the_repository (void ); +void prepare_alt_odb (struct repository *r);#endif /* OBJECT_STORE_H */
sha1_file.c +11 -10
@@ -677,21 +677,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb_the_repository (void ) +void prepare_alt_odb (struct repository *r) { -const char *alt; - -if (the_repository->objects.alt_odb_tail) +if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); + r->objects.alt_odb_tail = &r->objects.alt_odb_list; + +if (!r->ignore_env) { +const char *alt =getenv (ALTERNATE_DB_ENVIRONMENT ); +if (!alt) + alt ="" ; - the_repository->objects.alt_odb_tail = - &the_repository->objects.alt_odb_list; -link_alt_odb_entries (the_repository, alt, -PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); + } -read_info_alternates (the_repository,get_object_directory (),0 ); +read_info_alternates (r, r->objects.objectdir,0 ); }/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
Re: [PATCH 14/26] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
Quoted 31 lines · Stefan Beller
Signed-off-by Stefan Beller <[email protected]>--- object-store.h | 3 +-- sha1_file.c | 21 +++++++++++---------- 2 files changed, 12 insertions(+), 12 deletions(-)object-store.h +1
-2@@ -61,7 +61,6 @@ struct packed_git { char pack_name[FLEX_ARRAY]; /* more */ }; -#define prepare_alt_odb (r) prepare_alt_odb_##r() -extern void prepare_alt_odb_the_repository (void ); +void prepare_alt_odb (struct repository *r);#endif /* OBJECT_STORE_H */ sha1_file.c +1
-2@@ -677,21 +677,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb_the_repository (void ) +void prepare_alt_odb (struct repository *r) { -const char *alt;
sha1_file.c int foreach_alt_odb(alt_odb_fn fn, void *cb) · line 688
Quoted 10 lines
sha1_file.c +5
-3- -if (the_repository->objects.alt_odb_tail) +if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); + r->objects.alt_odb_tail = &r->objects.alt_odb_list; + +if (!r->ignore_env) { +const char *alt =getenv (ALTERNATE_DB_ENVIRONMENT );
If one day the majority of git moves to use 'struct repository', then ALTERNATE_DB_ENVIRONMENT is always ignored because ignore_env is always true. I think if you ignore_env, then you still need to get this "alt" from 'struct raw_object_store' (or 'struct repository').
Since you get lots of getenv() in repo_setup_env(), I think this getenv(ALTERNATE_DB_ENVIRONMENT) belongs there too. Then here, if ignore_env is true, you read r->objects.alt or something instead of doing getenv().
I really want to kill this getenv() in this code, which is basically delayed initialization because we haven't done proper init on the_repo. I realize that it cannot be done earlier, when prepare_alt_odb() does not even have a 'struct repository *' to work with. Would it be ok if I contributed a patch on top of your series to basically do repo_init(&the_repo) for all builtin/external commands (and fix all the bugs that come with it)? Then we would not need ignore_env here anymore.
Quoted 18 lines
sha1_file.c +5
-6+if (!alt) + alt ="" ; - the_repository->objects.alt_odb_tail = - &the_repository->objects.alt_odb_list; -link_alt_odb_entries (the_repository, alt, -PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); + } -read_info_alternates (the_repository,get_object_directory (),0 ); +read_info_alternates (r, r->objects.objectdir,0 ); }/* Returns 1 if we have successfully freshened the file, 0 otherwise. */ --2.16.1.73.ga2c3e9663f.dirty
Signature · 1 line
Duy
Re: [PATCH 14/26] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
Quoted 53 lines · Duy Nguyen
Quoted 31 lines · Stefan Beller
Signed-off-by Stefan Beller <[email protected]>--- object-store.h | 3 +-- sha1_file.c | 21 +++++++++++---------- 2 files changed, 12 insertions(+), 12 deletions(-)object-store.h +1
-2@@ -61,7 +61,6 @@ struct packed_git { char pack_name[FLEX_ARRAY]; /* more */ }; -#define prepare_alt_odb (r) prepare_alt_odb_##r() -extern void prepare_alt_odb_the_repository (void ); +void prepare_alt_odb (struct repository *r);#endif /* OBJECT_STORE_H */ sha1_file.c +1
-2@@ -677,21 +677,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb_the_repository (void ) +void prepare_alt_odb (struct repository *r) { -const char *alt;sha1_file.c int foreach_alt_odb(alt_odb_fn fn, void *cb) · line 688
Quoted 10 lines
sha1_file.c +5
-3- -if (the_repository->objects.alt_odb_tail) +if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); + r->objects.alt_odb_tail = &r->objects.alt_odb_list; + +if (!r->ignore_env) { +const char *alt =getenv (ALTERNATE_DB_ENVIRONMENT );If one day the majority of git moves to use 'struct repository', then
ALTERNATE_DB_ENVIRONMENTis always ignored becauseignore_envis always true. I think if youignore_env, then you still need to get this "alt" from 'structraw_object_store' (or 'struct repository').Since you get lots of
getenv()inrepo_setup_env(), I think thisgetenv(ALTERNATE_DB_ENVIRONMENT)belongs there too. Then here, ifignore_envis true, you readr->objects.altor something instead of doinggetenv().
Quoted 8 lines
I really want to kill this
getenv()in this code, which is basically delayed initialization because we haven't done proper init onthe_repo. I realize that it cannot be done earlier, whenprepare_alt_odb()does not even have a 'struct repository *' to work with. Would it be ok if I contributed a patch on top of your series to basically dorepo_init(&the_repo)for all builtin/external commands (and fix all the bugs that come with it)? Then we would not needignore_envhere anymore.
At some point yes we would definitely want the setup code to fully initialize a repository object (in this case the_repository). And I would even like to change the function signatures of all the builtin commands to take a repository object so that they don't implicitly rely on the_repository at all.
When I introduced struct repository I seem to remember there being a couple things which were different about setup that made it difficult to simply call repo_init() on the_repository. If you can fix whatever those issues with setup were (I can't remember all of them) then that would be great :)
Quoted 24 lines
Quoted 19 lines
sha1_file.c +5
-6+if (!alt) + alt ="" ; - the_repository->objects.alt_odb_tail = - &the_repository->objects.alt_odb_list; -link_alt_odb_entries (the_repository, alt, -PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); + } -read_info_alternates (the_repository,get_object_directory (),0 ); +read_info_alternates (r, r->objects.objectdir,0 ); }/* Returns 1 if we have successfully freshened the file, 0 otherwise. */ --2.16.1.73.ga2c3e9663f.dirty
Signature · 1 line
Duy
Signature · 1 line
Brandon Williams
cover Fix initializing the_hash_algo
Quoted 11 lines · Brandon Williams
At some point yes we would definitely want the setup code to fully initialize a repository object (in this case
the_repository). And I would even like to change the function signatures of all the builtin commands to take a repository object so that they don't implicitly rely onthe_repositoryat all.When I introduced struct repository I seem to remember there being a couple things which were different about setup that made it difficult to simply call
repo_init()onthe_repository. If you can fix whatever those issues with setup were (I can't remember all of them) then that would be great :)
I can certainly try! I start to remember all the hairy details in that setup code.
The first step may be something like this, which identifies all the "repo init" entry points. This is basically a revert of e26f7f19b6 (repository: pre-initialize hash algo pointer - 2018-01-19) and doing things the proper way, hopefully.
This is on 'master', independent from Stefan's series. I have another patch on top of that series to remove the use of ignore_env in sha1_file.c (and things seem to work). Basically whenever you have to initialize the hash algorithm, there's a good chance you need to initialize object store as well. But I'll hold that off until Stefan's and this one are both merged.
But yeah, it looks like we need some surgery in setup.c if we want something as pretty as repo_submodule_init() but for the main repo.
Nguyễn Thái Ngọc Duy (2):
setup.c: initialize the_repository correctly in all cases Revert "repository: pre-initialize hash algo pointer"
builtin/index-pack.c | 5 +++++ builtin/init-db.c | 3 ++- cache.h | 3 ++- common-main.c | 4 ++++ diff-no-index.c | 5 +++++ path.c | 2 +- repository.c | 2 +- setup.c | 5 ++++- t/helper/test-dump-split-index.c | 2 ++ 9 files changed, 26 insertions(+), 5 deletions(-)
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH 1/2 setup.c: initialize the_repository correctly in all cases
There are many ways for any command to access a git repository:
- most of them will try to discover the .git dir via setup_git_directory() and friends
- the server side programs already know where the repo is and prepare with enter_repo()
- special commands that deal with repo creation (init/clone) use init_db() once the new repo is ready for access.
- somebody accesses $GIT_DIR before any of above functions are called and accidentally sets up a git repository by set_git_dir() alone
"the_repository" is partially set up via set_git_dir() at some point in all four cases. The hash algorithm though is configured later after .git/config is read.
So far proper repo initialization is done only for the first case [1]. The second case is not covered (but that's fine [3]). The third case was found and worked around in [2]. The fourth case is a buggy one, which should be fixed already by jk/no-looking-at-dotgit-outside-repo and never happens again.
This patch makes sure all cases initialize the hash algorithm in the_repository correctly. Both second and third cases must run check_repo_format() before "entering" it. Eventually we probably just rename this function to init_repo() or something.
[1] 78a6766802 (Integrate hash algorithm support with repo setup -
2017-11-12)
[2] e26f7f19b6 (repository: pre-initialize hash algo pointer -
2018-01-19)
[3] the reason server side is still running ok with no hash algo before [2] is because the programs that use enter_repo() do very little then spawn a new program (like pack-objects or upload-archive) to do the heavy lifting. These programs already use setup_git_dir..()
--- builtin/init-db.c | 3 ++- cache.h | 3 ++- path.c | 2 +- setup.c | 5 ++++- 4 files changed, 9 insertions(+), 4 deletions(-)
builtin/init-db.c +2 -1
@@ -9,6 +9,7 @@#include "builtin.h" #include "exec_cmd.h" #include "parse-options.h" +#include "repository.h" #ifndef DEFAULT_GIT_TEMPLATE_DIR #define DEFAULT_GIT_TEMPLATE_DIR "/usr/share/git-core/templates"
@@ -369,7 +370,7 @@ int init_db(const char *git_dir, const char *real_git_dir,* config file, so this will not fail. What we are catching * is an attempt to reinitialize new repository with an old tool. */ -check_repository_format (); +check_repository_format (the_repository); reinit =create_default_files (template_dir, original_git_dir);
cache.h +2 -1
@@ -894,6 +894,7 @@ extern int repository_format_precious_objects;extern char *repository_format_partial_clone;extern const char *core_partial_clone_filter_default; +struct repository ;struct repository_format {int version;int precious_objects;
@@ -926,7 +927,7 @@ int verify_repository_format(const struct repository_format *format,* set_git_dir() before calling this, and use it only for "are we in a valid * repo?". */ -extern void check_repository_format (void ); +extern void check_repository_format (struct repository *);#define MTIME_CHANGED 0x0001#define CTIME_CHANGED 0x0002
path.c +1 -1
@@ -827,7 +827,7 @@ const char *enter_repo(const char *path, int strict)if (is_git_directory ("." )) {set_git_dir ("." ); -check_repository_format (); +check_repository_format (the_repository);return path; }
setup.c +4 -1
@@ -1180,11 +1180,14 @@ int git_config_perm(const char *var, const char *value)return -(i &0666 ); } -void check_repository_format (void ) +/* optionally configure "repo" to the correct format */ +void check_repository_format (struct repository *repo) {struct repository_format repo_fmt;check_repository_format_gently (get_git_dir (), &repo_fmt,NULL ); startup_info->have_repository =1 ; +if (repo) +repo_set_hash_algo (repo, repo_fmt.hash_algo); }/*
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH 1/2] setup.c: initialize the_repository correctly in all cases
Quoted 5 lines · Nguyễn Thái Ngọc Duy
[3] the reason server side is still running ok with no hash algo before [2] is because the programs that use
enter_repo()do very little then spawn a new program (like pack-objects or upload-archive) to do the heavy lifting. These programs already use setup_git_dir..()
You have "..()" here. Did you want to say "()." instead?
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
Re: [PATCH 1/2] setup.c: initialize the_repository correctly in all cases
Quoted 8 lines · brian m. carlson
Quoted 5 lines · Nguyễn Thái Ngọc Duy
[3] the reason server side is still running ok with no hash algo before [2] is because the programs that use
enter_repo()do very little then spawn a new program (like pack-objects or upload-archive) to do the heavy lifting. These programs already use setup_git_dir..()You have "..()" here. Did you want to say "()." instead?
It's because we have two functions, setup_git_directory() and setup_git_directory_gently() and I tend to just put '*' instead of listing both. But since it causes confusion, I'm listing both in v2.
Signature · 1 line
Duy
PATCH 2/2 Revert "repository: pre-initialize hash algo pointer"
In reply to Nguyễn Thái Ngọc DuyThis reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root problem, git clone not setting up the_hash_algo, has been fixed in the previous patch.
As a result of the revert, some code paths that use the_hash_algo without initialization is revealed and fixed here. It's basically commands that are allowed to run without a repository. The fix here is not the best. We probably could figure out the hash algorithm from input somehow.
Since this is a dangerous move and could potentially break stuff after release (and leads to workaround like the reverted commit), the workaround technically remains, but is hidden behind a new environment variable GIT_HASH_FIXUP. This should let the users continue to use git while we fix the problem.
--- builtin/index-pack.c | 5 +++++ common-main.c | 4 ++++ diff-no-index.c | 5 +++++ repository.c | 2 +- t/helper/test-dump-split-index.c | 2 ++ 5 files changed, 17 insertions(+), 1 deletion(-)
builtin/index-pack.c +5 -0
@@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)if (prefix &&chdir (prefix))die (_ ("Cannot come back to cwd" )); +if (!the_hash_algo) { +warning (_ ("Running without a repository, assuming SHA-1 hash" )); +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); + } +for (i =1 ; i < argc; i++) {const char *arg = argv[i];
common-main.c +4 -0
@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the
@@ -40,5 +41,8 @@ int main(int argc, const char **argv)restore_sigpipe_to_default (); +if (getenv ("GIT_HASH_FIXUP" )) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +return cmd_main (argc, argv); }
diff-no-index.c +5 -0
@@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,struct strbuf replacement =STRBUF_INIT ;const char *prefix = revs->prefix; +if (!the_hash_algo) { +warning (_ ("Running without a repository, assuming SHA-1 hash" )); +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); + } +diff_setup (&revs->diffopt);for (i =1 ; i < argc -2 ; ) {int j;
repository.c +1 -1
@@ -5,7 +5,7 @@/* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index,NULL ,0 ,0 };struct repository *the_repository = &the_repo;
t/helper/test-dump-split-index.c +2 -0
@@ -12,6 +12,8 @@ int cmd_main(int ac, const char **av)struct split_index *si;int i; +setup_git_directory (); +do_read_index (&the_index, av[1 ],1 );printf ("own %s\n" ,sha1_to_hex (the_index.sha1)); si = the_index.split_index;
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
Quoted 83 lines · Nguyễn Thái Ngọc Duy
This reverts commit
e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root problem, git clone not setting upthe_hash_algo, has been fixed in the previous patch.As a result of the revert, some code paths that use
the_hash_algowithout initialization is revealed and fixed here. It's basically commands that are allowed to run without a repository. The fix here is not the best. We probably could figure out the hash algorithm from input somehow.Since this is a dangerous move and could potentially break stuff after release (and leads to workaround like the reverted commit), the workaround technically remains, but is hidden behind a new environment variable
GIT_HASH_FIXUP. This should let the users continue to use git while we fix the problem.Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>--- builtin/index-pack.c | 5 +++++ common-main.c | 4 ++++ diff-no-index.c | 5 +++++ repository.c | 2 +- t/helper/test-dump-split-index.c | 2 ++ 5 files changed, 17 insertions(+), 1 deletion(-)builtin/index-pack.c +5
-0@@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)if (prefix &&chdir (prefix))die (_ ("Cannot come back to cwd" )); +if (!the_hash_algo) { +warning (_ ("Running without a repository, assuming SHA-1 hash" )); +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); + } +for (i =1 ; i < argc; i++) {const char *arg = argv[i];common-main.c +4
-0@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the @@ -40,5 +41,8 @@ int main(int argc, const char **argv)restore_sigpipe_to_default (); +if (getenv ("GIT_HASH_FIXUP" )) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +return cmd_main (argc, argv); }diff-no-index.c +5
-0@@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,struct strbuf replacement =STRBUF_INIT ;const char *prefix = revs->prefix; +if (!the_hash_algo) { +warning (_ ("Running without a repository, assuming SHA-1 hash" )); +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); + } +diff_setup (&revs->diffopt);for (i =1 ; i < argc -2 ; ) {int j;repository.c +0
-0
repository.c line 8
Quoted 5 lines
repository.c +1
-1@@ -5,7 +5,7 @@/* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index,NULL ,0 ,0
I am not sure I agree with this defense in depth, because it would add a lot to maintenance burden.
Specifically this part. The series that I sent out usually clashes here as this is currently a hot area
of the code touched by many different series in flight.
However this is the long term correct thing to do? We assume no algorithm until the repository can tell us from its config (or we default to sha1 if there is no configuration present).
I wonder if there is yet another missing case in the enumeration of the previous patch:
Some commands are able to operate on GIT_OBJECT_DIR instead
of GIT_DIR (git repack?), which may not even explore the full git directory, and so doesn't know about the hash value.
In the cover letter you reference my series, but comparing the diffstats (and looking through the patches), I would only expect this one place to have merge conflicts, which ought to be easy to resolve. (In my series I break the initializer into multiple lines to help the future, too)
After some thought, I like this series.
Thanks,
Stefan
Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
Quoted 5 lines · Stefan Beller
I wonder if there is yet another missing case in the enumeration of the previous patch:
Some commands are able to operate onGIT_OBJECT_DIRinstead
ofGIT_DIR(git repack?), which may not even explore the full git directory, and so doesn't know about the hash value.
... because GIT_DIR/config is not known? "repack" is not one of them, though---it needs to at least use refs as the starting point so a standalone OBJECT_DIR is insufficient.
Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
Quoted 11 lines · Junio C Hamano
Quoted 5 lines · Stefan Beller
I wonder if there is yet another missing case in the enumeration of the previous patch:
Some commands are able to operate onGIT_OBJECT_DIRinstead
ofGIT_DIR(git repack?), which may not even explore the full git directory, and so doesn't know about the hash value.... because GIT_DIR/config is not known? "repack" is not one of them, though---it needs to at least use refs as the starting point so a standalone
OBJECT_DIRis insufficient.
Yes, I could have worded this as a question:
Is there any command that operates on GIT_OBJECT_DIR without trying to discover GIT_DIR ?
Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
Quoted 9 lines · Stefan Beller
Quoted 8 lines · Junio C Hamano
Quoted 5 lines · Stefan Beller
I wonder if there is yet another missing case in the enumeration of the previous patch:
Some commands are able to operate onGIT_OBJECT_DIRinstead
ofGIT_DIR(git repack?), which may not even explore the full git directory, and so doesn't know about the hash value.
Quoted 7 lines
Quoted 3 lines
... because GIT_DIR/config is not known? "repack" is not one of them, though---it needs to at least use refs as the starting point so a standalone
OBJECT_DIRis insufficient.Yes, I could have worded this as a question:
Is there any command that operates onGIT_OBJECT_DIRwithout trying to discoverGIT_DIR?
If somebody finds one that would be a good argument not to pursue the approach. Lack of response to the question would not amount to that much---it is possible all people who bothered to think of overlooked something obvious, though. "When I asked around nobody thought of a possibly way for this to cause breakages, so let's declare it is safe to do so and do it" is not how we want to do things X-<.
Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
In reply to Nguyễn Thái Ngọc Duybuiltin/index-pack.c int cmd_index_pack(int argc, const char **argv, const char *prefix) · line 1679
Quoted 12 lines · Nguyễn Thái Ngọc Duy
builtin/index-pack.c +4
-0@@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)if (prefix &&chdir (prefix))die (_ ("Cannot come back to cwd" )); +if (!the_hash_algo) { +warning (_ ("Running without a repository, assuming SHA-1 hash" )); +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); + }
Is this warning going to be visible to users in the normal course of operation? If so, people are probably going to find this bothersome or alarming.
Quoted 16 lines
builtin/index-pack.c +0
-0for (i =1 ; i < argc; i++) {const char *arg = argv[i];common-main.c +1
-0@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the
common-main.c int main(int argc, const char **argv) · line 45
Quoted 5 lines
common-main.c +2
-0@@ -40,5 +41,8 @@ int main(int argc, const char **argv)restore_sigpipe_to_default (); +if (getenv ("GIT_HASH_FIXUP" )) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 );
I'm lukewarm on adding this environment variable, but considering our history here, we had probably better. We can always remove it after a few releases.
Quoted 7 lines
common-main.c +0
-0return cmd_main (argc, argv); }diff-no-index.c +0
-0
diff-no-index.c void diff_no_index(struct rev_info *revs, · line 247
Quoted 7 lines
diff-no-index.c +4
-0@@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,struct strbuf replacement =STRBUF_INIT ;const char *prefix = revs->prefix; +if (!the_hash_algo) { +warning (_ ("Running without a repository, assuming SHA-1 hash" )); +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); + }
Again, same concern. I can imagine scripts that will blow up loudly if git diff --no-index spews things to standard error.
I'm not opposed to making this more visible, but I wonder if maybe it should only be visible to developers or such. The only way I can think of doing is that is with an advice options, but maybe there's a better way.
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
Quoted 35 lines · brian m. carlson
builtin/index-pack.c int cmd_index_pack(int argc, const char **argv, const char *prefix) · line 1679
Quoted 12 lines · Nguyễn Thái Ngọc Duy
builtin/index-pack.c +4
-0@@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)if (prefix &&chdir (prefix))die (_ ("Cannot come back to cwd" )); +if (!the_hash_algo) { +warning (_ ("Running without a repository, assuming SHA-1 hash" )); +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); + }Is this warning going to be visible to users in the normal course of operation? If so, people are probably going to find this bothersome or alarming.
Quoted 17 lines
builtin/index-pack.c +0
-0for (i =1 ; i < argc; i++) {const char *arg = argv[i];common-main.c +1
-0@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the @@ -40,5 +41,8 @@ int main(int argc, const char **argv)
Quoted 8 lines
common-main.c int main(int argc, const char **argv) · line 45
Quoted 4 lines
common-main.c +2
-0restore_sigpipe_to_default (); +if (getenv ("GIT_HASH_FIXUP" )) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 );I'm lukewarm on adding this environment variable, but considering our history here, we had probably better. We can always remove it after a few releases.
Yeah I don't know what the best thing to do here would be. I mean we could always just init the hash_algo for the_repository here so that we don't ever have to worry about it, but I don't know if even that is the right approach.
Quoted 27 lines
Quoted 8 lines
common-main.c +0
-0return cmd_main (argc, argv); }diff-no-index.c +0
-0diff-no-index.c void diff_no_index(struct rev_info *revs, · line 247
Quoted 7 lines
diff-no-index.c +4
-0@@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,struct strbuf replacement =STRBUF_INIT ;const char *prefix = revs->prefix; +if (!the_hash_algo) { +warning (_ ("Running without a repository, assuming SHA-1 hash" )); +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); + }Again, same concern. I can imagine scripts that will blow up loudly if git diff --no-index spews things to standard error.
I'm not opposed to making this more visible, but I wonder if maybe it should only be visible to developers or such. The only way I can think of doing is that is with an advice options, but maybe there's a better way.
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
Signature · 1 line
Brandon Williams
Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
In reply to brian m. carlsonQuoted 10 lines · brian m. carlson
common-main.c int main(int argc, const char **argv) · line 45
Quoted 6 lines
common-main.c +2
-0@@ -40,5 +41,8 @@ int main(int argc, const char **argv)restore_sigpipe_to_default (); +if (getenv ("GIT_HASH_FIXUP" )) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 );I'm lukewarm on adding this environment variable, but considering our history here, we had probably better. We can always remove it after a few releases.
Yes that's the intention. But after writing cover letter for v2 and sending it out, it looks to me that this thing must stay until all our code is converted to using the_hash_algo (I don't know if there are more to convert or it's finished already). So an alternative is we do the opposite: default to GIT_HASH_SHA1, but when an env variable is set, reset it back to NULL. This env variable will _always_ be set by the test suite to help us catch problems.
Signature · 1 line
Duy
Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
Quoted 7 lines · Duy Nguyen
Yes that's the intention. But after writing cover letter for v2 and sending it out, it looks to me that this thing must stay until all our code is converted to using
the_hash_algo(I don't know if there are more to convert or it's finished already). So an alternative is we do the opposite: default toGIT_HASH_SHA1, but when an env variable is set, reset it back to NULL. This env variable will _always_ be set by the test suite to help us catch problems.
If we were to do the "do not blindly initialize and always initialize explicitly for debugging" (which I doubt is a good direction to go in), then what you outlined above certainly is a less evil way to do so than what the patch under discussion did, I would think.
Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
Quoted 10 lines · Junio C Hamano
Quoted 7 lines · Duy Nguyen
Yes that's the intention. But after writing cover letter for v2 and sending it out, it looks to me that this thing must stay until all our code is converted to using
the_hash_algo(I don't know if there are more to convert or it's finished already). So an alternative is we do the opposite: default toGIT_HASH_SHA1, but when an env variable is set, reset it back to NULL. This env variable will _always_ be set by the test suite to help us catch problems.
Quoted 5 lines
If we were to do the "do not blindly initialize and always initialize explicitly for debugging" (which I doubt is a good direction to go in), then what you outlined above certainly is a less evil way to do so than what the patch under discussion did, I would think.
So v3 [1] should be in good condition to go to 'pu', I think?
[1] https://public-inbox.org/git/[email protected]/
Signature · 1 line
Duy
Re: [PATCH 0/2] Fix initializing the_hash_algo
In reply to Nguyễn Thái Ngọc DuyQuoted 8 lines · Nguyễn Thái Ngọc Duy
I can certainly try! I start to remember all the hairy details in that setup code.
The first step may be something like this, which identifies all the "repo init" entry points. This is basically a revert of
e26f7f19b6(repository: pre-initialize hash algo pointer - 2018-01-19) and doing things the proper way, hopefully.
Quoted 6 lines
This is on 'master', independent from Stefan's series. I have another patch on top of that series to remove the use of
ignore_envinsha1_file.c(and things seem to work). Basically whenever you have to initialize the hash algorithm, there's a good chance you need to initialize object store as well. But I'll hold that off until Stefan's and this one are both merged.
I definitely think this series is an improvement over my previous patch.
My major concern is alarming users (or breaking scripts) with the warning message. I wonder if deferring the use of the warning until we have multiple hash algorithms might be a better idea. At that point, the warning would become something people could act upon.
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
v2 cover Fix initializing the_hash_algo
In reply to Nguyễn Thái Ngọc DuyBrian questioned the unnecessary alarms in v1 when "git diff" or "git index-pack" run in no-repository mode. I had the same feeling after sending v1. But instead of suppresing alarms, we could do better.
v2 breaks those "fall back to SHA-1" code into separate patches and handles it properly (I hope) instead of blind fall back like v1:
- for index-pack, we can determine the needed algorithm from the pack file. I'm making an assumption here that pack files with new hash algo must step up file format version. But I think it's a reasonable assumption.
- for diff --no-index, I still fall back to SHA-1 to generate the hashes like before. We could probably introduce a new command line option to use a different hash. But that work could be done later when an actual new hash has come
Note, I didn't test but this series could potentially break 'pu' (in a good way). I initially was puzzled why the test suite didn't fail when the_repository->hash_algo was NULL (i.e. before Brian's fix). Then I found out that more work to abstract away SHA-1 hashing functions have been done since then. But because the_hash_algo is pre-initialized with SHA-1, these special cases did not show up until now. If there are more the_hash_algo conversion on 'pu', more "no-repo" cases could be spotted by the test suite.
I think this makes pre-initializing the_hash_algo to NULL a very good
point: during the abstraction work, we at least must identify the use cases like this (code running without repo). We don't have to fix it right away, but we can start thinking about how to deal with it. If we ignore it and switch to a new hash, these code will keep using SHA-1 and cause more headache in future.
Nguyễn Thái Ngọc Duy (5):
setup.c: initialize the_repository correctly in all cases sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN]
index-pack: check (and optionally set) hash algo based on input file diff.c: initialize hash algo when running in --no-index mode Revert "repository: pre-initialize hash algo pointer"
builtin/index-pack.c | 26 +++++++++++++++++++++++++- builtin/init-db.c | 3 ++- cache.h | 3 ++- common-main.c | 10 ++++++++++ diff.c | 12 ++++++++++++ path.c | 2 +- repository.c | 2 +- setup.c | 5 ++++- sha1_file.c | 2 +- t/helper/test-dump-split-index.c | 2 ++ 10 files changed, 60 insertions(+), 7 deletions(-)
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH v2 1/5 setup.c: initialize the_repository correctly in all cases
There are many ways for any command to access a git repository:
- most of them will try to discover the .git dir via setup_git_directory() and friends
- the server side programs already know where the repo is and prepare with enter_repo()
- special commands that deal with repo creation (init/clone) use init_db() once the new repo is ready for access.
- somebody accesses $GIT_DIR before any of above functions are called and accidentally sets up a git repository by set_git_dir() alone
"the_repository" is partially set up via set_git_dir() at some point in all four cases. The hash algorithm though is configured later after .git/config is read.
So far proper repo initialization is done only for the first case [1]. The second case is not covered (but that's fine [3]). The third case was found and worked around in [2]. The fourth case is a buggy one, which should be fixed already by jk/no-looking-at-dotgit-outside-repo and never happens again.
This patch makes sure all cases initialize the hash algorithm in the_repository correctly. Both second and third cases must run check_repo_format() before "entering" it. Eventually we probably just rename this function to init_repo() or something.
[1] 78a6766802 (Integrate hash algorithm support with repo setup -
2017-11-12)
[2] e26f7f19b6 (repository: pre-initialize hash algo pointer -
2018-01-19)
[3] the reason server side is still running ok with no hash algo before
[2] is because the programs that use enter_repo() do very little (and unlikely to hash anything) then spawn a new program (like pack-objects or upload-archive) to do the heavy lifting. These programs already use setup_git_directory() or the gently version
--- builtin/init-db.c | 3 ++- cache.h | 3 ++- path.c | 2 +- setup.c | 5 ++++- 4 files changed, 9 insertions(+), 4 deletions(-)
builtin/init-db.c +2 -1
@@ -9,6 +9,7 @@#include "builtin.h" #include "exec_cmd.h" #include "parse-options.h" +#include "repository.h" #ifndef DEFAULT_GIT_TEMPLATE_DIR #define DEFAULT_GIT_TEMPLATE_DIR "/usr/share/git-core/templates"
@@ -369,7 +370,7 @@ int init_db(const char *git_dir, const char *real_git_dir,* config file, so this will not fail. What we are catching * is an attempt to reinitialize new repository with an old tool. */ -check_repository_format (); +check_repository_format (the_repository); reinit =create_default_files (template_dir, original_git_dir);
cache.h +2 -1
@@ -894,6 +894,7 @@ extern int repository_format_precious_objects;extern char *repository_format_partial_clone;extern const char *core_partial_clone_filter_default; +struct repository ;struct repository_format {int version;int precious_objects;
@@ -926,7 +927,7 @@ int verify_repository_format(const struct repository_format *format,* set_git_dir() before calling this, and use it only for "are we in a valid * repo?". */ -extern void check_repository_format (void ); +extern void check_repository_format (struct repository *);#define MTIME_CHANGED 0x0001#define CTIME_CHANGED 0x0002
path.c +1 -1
@@ -827,7 +827,7 @@ const char *enter_repo(const char *path, int strict)if (is_git_directory ("." )) {set_git_dir ("." ); -check_repository_format (); +check_repository_format (the_repository);return path; }
setup.c +4 -1
@@ -1180,11 +1180,14 @@ int git_config_perm(const char *var, const char *value)return -(i &0666 ); } -void check_repository_format (void ) +/* optionally configure "repo" to the correct format */ +void check_repository_format (struct repository *repo) {struct repository_format repo_fmt;check_repository_format_gently (get_git_dir (), &repo_fmt,NULL ); startup_info->have_repository =1 ; +if (repo) +repo_set_hash_algo (repo, repo_fmt.hash_algo); }/*
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH v2 2/5 sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN]
In reply to Nguyễn Thái Ngọc DuyThis is mostly for displaying the hash algorithm name when we report errors. Printing "unknown" with '%s' is much better than '(null)' in glibc printf version (and probably could crash if other implementations do not check for NULL pointer)
--- sha1_file.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
sha1_file.c +1 -1
@@ -71,7 +71,7 @@ static void git_hash_unknown_final(unsigned char *hash, git_hash_ctx *ctx)const struct git_hash_algo hash_algos[GIT_HASH_NALGOS ] = { { -NULL , +"unknown" ,0x00000000 ,0 ,0 ,
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH v2 2/5] sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN]
Quoted 4 lines · Nguyễn Thái Ngọc Duy
This is mostly for displaying the hash algorithm name when we report errors. Printing "unknown" with '%s' is much better than '(null)' in glibc printf version (and probably could crash if other implementations do not check for NULL pointer)
This is a good change. Using a NULL pointer with %s on the BSDs definitely did cause a crash, last I checked.
I looked at the comment I wrote in hash.h, and based on that, it is indeed very likely that someone would accidentally write code this way.
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
PATCH v2 3/5 index-pack: check (and optionally set) hash algo based on input file
In reply to Nguyễn Thái Ngọc DuyAfter 454253f059 (builtin/index-pack: improve hash function abstraction
- 2018-02-01), index-pack uses the_hash_algo for hashing. If "git index-pack" is executed without a repository, we do not know what hash algorithm to be used and the_hash_algo in theory could be undefined.
Since there should be some information about the hash algorithm in the input pack file, we can initialize the correct hash algorithm with that if the_hash_algo is not yet initialized. This assumes that pack files with new hash algorithm MUST step up pack version.
While at there, make sure the hash algorithm requested by the pack file and configured by the repository (if we're running with a repo) are consistent.
--- builtin/index-pack.c | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-)
builtin/index-pack.c +25 -1
@@ -326,10 +326,31 @@ static const char *open_pack_file(const char *pack_name) output_fd =-1 ; nothread_data.pack_fd = input_fd; } - the_hash_algo->init_fn (&input_ctx);return pack_name; } +static void prepare_hash_algo (uint32_t pack_version) +{ +const struct git_hash_algo *pack_algo; + +switch (pack_version) { +case 2 : +case 3 : + pack_algo = &hash_algos[GIT_HASH_SHA1 ]; +break ; +default : +die ("BUG: how to determine hash algo for new version?" ); + } + +if (!the_hash_algo)/* running without repo */ + the_hash_algo = pack_algo; + +if (the_hash_algo != pack_algo) +die (_ ("incompatible hash algorithm, " +"configured for %s but the pack file needs %s" ), + the_hash_algo->name, pack_algo->name); +} +static void parse_pack_header (void ) {struct pack_header *hdr =fill (sizeof (struct pack_header ));
@@ -341,6 +362,9 @@ static void parse_pack_header(void)die (_ ("pack version %" PRIu32" unsupported" ),ntohl (hdr->hdr_version)); +prepare_hash_algo (ntohl (hdr->hdr_version)); + the_hash_algo->init_fn (&input_ctx); + nr_objects =ntohl (hdr->hdr_entries);use (sizeof (struct pack_header )); }
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH v2 3/5] index-pack: check (and optionally set) hash algo based on input file
Quoted 41 lines · Nguyễn Thái Ngọc Duy
After
454253f059(builtin/index-pack: improve hash function abstraction
- 2018-02-01), index-pack usesthe_hash_algofor hashing. If "git index-pack" is executed without a repository, we do not know what hash algorithm to be used andthe_hash_algoin theory could be undefined.Since there should be some information about the hash algorithm in the input pack file, we can initialize the correct hash algorithm with that if
the_hash_algois not yet initialized. This assumes that pack files with new hash algorithm MUST step up pack version.While at there, make sure the hash algorithm requested by the pack file and configured by the repository (if we're running with a repo) are consistent.
Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>--- builtin/index-pack.c | 26 +++++++++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-)builtin/index-pack.c +10
-1@@ -326,10 +326,31 @@ static const char *open_pack_file(const char *pack_name) output_fd =-1 ; nothread_data.pack_fd = input_fd; } - the_hash_algo->init_fn (&input_ctx);return pack_name; } +static void prepare_hash_algo (uint32_t pack_version) +{ +const struct git_hash_algo *pack_algo; + +switch (pack_version) { +case 2 : +case 3 : + pack_algo = &hash_algos[GIT_HASH_SHA1 ]; +break ; +default :
builtin/index-pack.c static const char *open_pack_file(const char *pack_name) · line 351
Quoted 10 lines
builtin/index-pack.c +10
-0+die ("BUG: how to determine hash algo for new version?" ); + } + +if (!the_hash_algo)/* running without repo */ + the_hash_algo = pack_algo; + +if (the_hash_algo != pack_algo) +die (_ ("incompatible hash algorithm, " +"configured for %s but the pack file needs %s" ), + the_hash_algo->name, pack_algo->name);
I like this. It's a nice improvement and it should be easy for us to pass additional information into the function when our pack format understands multiple algorithms.
I might have done the comparison using the format_id members instead of the pointers themselves, but that's more a personal preference than anything.
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
PATCH v2 4/5 diff.c: initialize hash algo when running in --no-index mode
In reply to Nguyễn Thái Ngọc DuyOur "git diff" command supports running as a standalone tool. In this code path, we try to hash the file content but after 18e2588e11 (sha1_file: switch uses of SHA-1 to the_hash_algo - 2018-02-01), there is a chance that the_hash_algo (required by index_path) may still be uninitialized if no repository is found.
Executing index_path() when the_hash_algo is NULL (or points to unknown algo) either crashes or dies. Let's make it a bit safer by explicitly falling back to SHA-1 (so that the diff output remains the same as before, compared to the alternative that we simply do not hash).
dòng được
--- diff.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff.c +12 -0
@@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)return ; } +/* +* NEEDSWORK: When running in no-index mode (and no repo is +* found, thus no hash algo conifugred), fall back to SHA-1 +* hashing (which is used by diff_fill_oid_info below) to +* avoid regression in diff output. +* +* In future, perhaps we can allow the user to specify their +* hash algorithm from command line in this mode. +*/ +if (o->flags.no_index && !the_hash_algo) + the_hash_algo = &hash_algos[GIT_HASH_SHA1 ]; +diff_fill_oid_info (one);diff_fill_oid_info (two);
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH v2 4/5] diff.c: initialize hash algo when running in --no-index mode
Quoted 6 lines · Nguyễn Thái Ngọc Duy
Our "git diff" command supports running as a standalone tool. In this code path, we try to hash the file content but after
18e2588e11(sha1_file: switch uses of SHA-1 tothe_hash_algo- 2018-02-01), there is a chance thatthe_hash_algo(required byindex_path) may still be uninitialized if no repository is found.
Quoted 6 lines
Executing
index_path()whenthe_hash_algois NULL (or points to unknown algo) either crashes or dies. Let's make it a bit safer by explicitly falling back to SHA-1 (so that the diff output remains the same as before, compared to the alternative that we simply do not hash).dòng được
Accidental paste?
diff.c static void run_diff(struct diff_filepair *p, struct diff_options *o) · line 3997
Quoted 7 lines
Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>---
diff.c +3
-0@@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o) +/* +* NEEDSWORK: When running in no-index mode (and no repo is +* found, thus no hash algo conifugred), fall back to SHA-1
s/conifugred/configured/
Quoted 11 lines
diff.c +9
-0+* hashing (which is used by diff_fill_oid_info below) to +* avoid regression in diff output. +* +* In future, perhaps we can allow the user to specify their +* hash algorithm from command line in this mode. +*/ +if (o->flags.no_index && !the_hash_algo) + the_hash_algo = &hash_algos[GIT_HASH_SHA1 ]; +diff_fill_oid_info (one);diff_fill_oid_info (two);
Re: [PATCH v2 4/5] diff.c: initialize hash algo when running in --no-index mode
Quoted 7 lines · Eric Sunshine
Quoted 3 lines · Nguyễn Thái Ngọc Duy
before, compared to the alternative that we simply do not hash).
dòng được
Accidental paste?
Oops. I blame Emacs.
Signature · 1 line
Duy
Re: [PATCH v2 4/5] diff.c: initialize hash algo when running in --no-index mode
In reply to Nguyễn Thái Ngọc DuyQuoted 5 lines · Nguyễn Thái Ngọc Duy
diff.c +1
-0@@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)return ; } +/*
diff.c static void run_diff(struct diff_filepair *p, struct diff_options *o) · line 4008
Quoted 10 lines
diff.c +10
-0+* NEEDSWORK: When running in no-index mode (and no repo is +* found, thus no hash algo conifugred), fall back to SHA-1 +* hashing (which is used by diff_fill_oid_info below) to +* avoid regression in diff output. +* +* In future, perhaps we can allow the user to specify their +* hash algorithm from command line in this mode. +*/ +if (o->flags.no_index && !the_hash_algo) + the_hash_algo = &hash_algos[GIT_HASH_SHA1 ];
Brian, are we supposed to use the_hash_algo this way (i.e. as a writable var)? Or should I stick to something like
repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
which allows us to notify other parts inside struct repository about the hash algorithm change, if we ever need to?
If the_hash_algo is supposed to be read-only, maybe I should convert that macro to an inline function to prevent people from accidentally reassigning it?
Quoted 7 lines
diff.c +1
-1+diff_fill_oid_info (one);diff_fill_oid_info (two); --2.16.1.435.g8f24da2e1a
Signature · 1 line
Duy
Re: [PATCH v2 4/5] diff.c: initialize hash algo when running in --no-index mode
Quoted 21 lines · Duy Nguyen
Quoted 5 lines · Nguyễn Thái Ngọc Duy
diff.c +1
-0@@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)return ; } +/* diff.c static void run_diff(struct diff_filepair *p, struct diff_options *o) · line 4008
Quoted 10 lines
diff.c +10
-0+* NEEDSWORK: When running in no-index mode (and no repo is +* found, thus no hash algo conifugred), fall back to SHA-1 +* hashing (which is used by diff_fill_oid_info below) to +* avoid regression in diff output. +* +* In future, perhaps we can allow the user to specify their +* hash algorithm from command line in this mode. +*/ +if (o->flags.no_index && !the_hash_algo) + the_hash_algo = &hash_algos[GIT_HASH_SHA1 ];Brian, are we supposed to use
the_hash_algothis way (i.e. as a writable var)? Or should I stick to something like
Quoted 4 lines
repo_set_hash_algo(the_repository, GIT_HASH_SHA1);which allows us to notify other parts inside struct repository about the hash algorithm change, if we ever need to?
I would definitely recommend using the function. As you pointed out, it makes our code future-proof against needing to more work when setting the value.
Quoted 3 lines
If
the_hash_algois supposed to be read-only, maybe I should convert that macro to an inline function to prevent people from accidentally reassigning it?
You could if you want to, although I don't really see a need to, since we can just catch it in review. If you wanted to, I'd make it an inline function for performance reasons.
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
PATCH v2 5/5 Revert "repository: pre-initialize hash algo pointer"
In reply to Nguyễn Thái Ngọc DuyThis reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root problem, git clone not setting up the_hash_algo, has been fixed in the previous patch.
Since this is a dangerous move and could potentially break stuff after release (and leads to workaround like the reverted commit), the workaround technically remains, but is hidden behind a new environment variable GIT_HASH_FIXUP. This should let the users continue to use git while we fix the problem. This variable can be deleted after one or two releases.
--- common-main.c | 10 ++++++++++ repository.c | 2 +- t/helper/test-dump-split-index.c | 2 ++ 3 files changed, 13 insertions(+), 1 deletion(-)
common-main.c +10 -0
@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the
@@ -40,5 +41,14 @@ int main(int argc, const char **argv)restore_sigpipe_to_default (); +/* +* Temporary recovery measure while hashing code is being +* refactored. This variable should be gone after everybody +* has used the_hash_algo in one or two releases and nobody +* complains anything. +*/ +if (getenv ("GIT_HASH_FIXUP" )) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +return cmd_main (argc, argv); }
repository.c +1 -1
@@ -5,7 +5,7 @@/* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index,NULL ,0 ,0 };struct repository *the_repository = &the_repo;
t/helper/test-dump-split-index.c +2 -0
@@ -12,6 +12,8 @@ int cmd_main(int ac, const char **av)struct split_index *si;int i; +setup_git_directory (); +do_read_index (&the_index, av[1 ],1 );printf ("own %s\n" ,sha1_to_hex (the_index.sha1)); si = the_index.split_index;
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH v2 5/5] Revert "repository: pre-initialize hash algo pointer"
Quoted 50 lines · Nguyễn Thái Ngọc Duy
This reverts commit
e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root problem, git clone not setting upthe_hash_algo, has been fixed in the previous patch.Since this is a dangerous move and could potentially break stuff after release (and leads to workaround like the reverted commit), the workaround technically remains, but is hidden behind a new environment variable
GIT_HASH_FIXUP. This should let the users continue to use git while we fix the problem. This variable can be deleted after one or two releases.Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>--- common-main.c | 10 ++++++++++ repository.c | 2 +- t/helper/test-dump-split-index.c | 2 ++ 3 files changed, 13 insertions(+), 1 deletion(-)common-main.c +10
-0@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the @@ -40,5 +41,14 @@ int main(int argc, const char **argv)restore_sigpipe_to_default (); +/* +* Temporary recovery measure while hashing code is being +* refactored. This variable should be gone after everybody +* has used the_hash_algo in one or two releases and nobody +* complains anything. +*/ +if (getenv ("GIT_HASH_FIXUP" )) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +return cmd_main (argc, argv); }repository.c +0
-0
repository.c line 8
Quoted 5 lines
repository.c +1
-1@@ -5,7 +5,7 @@/* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index,NULL ,0 ,0
I'm wondering, now that you have the name field for the unknown value, if that might be a better choice here than NULL. I don't have a strong preference either way, so whatever you decide here is fine.
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
Re: [PATCH v2 5/5] Revert "repository: pre-initialize hash algo pointer"
Quoted 6 lines · brian m. carlson
Quoted 6 lines
repository.c +0
-0@@ -5,7 +5,7 @@
Quoted 8 lines
repository.c line 8
Quoted 4 lines
repository.c +1
-1/* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index,NULL ,0 ,0 I'm wondering, now that you have the name field for the unknown value, if that might be a better choice here than NULL. I don't have a strong preference either way, so whatever you decide here is fine.
I did try that first, but for the purpose of catching uninitialized algo use, NULL is better.
When I set unknown hash algo here, I think some test failed mysteriously because it used rawsz field (which has value zero), it didn't match some expectation, the code went to the error handling path, which eventually failed with some error message, but it's not obvious that the problem was rawsz being zero and back tracking that took me some time.
With NULL hash_algo, any dereferencing fails immediately with a nice stack trace. Another reason to push me towards NULL hash algo is, even if we prefer nice messages over segmentation faults, we can't avoid it completely anyway (empty_tree and empty_blob are still NULL in unknown hash algo and will cause segfaults). Might as well make things consistent and always segfault.
Signature · 1 line
Duy
Re: [PATCH v2 5/5] Revert "repository: pre-initialize hash algo pointer"
Quoted 7 lines · Duy Nguyen
When I set unknown hash algo here, I think some test failed mysteriously because it used rawsz field (which has value zero), it didn't match some expectation, the code went to the error handling path, which eventually failed with some error message, but it's not obvious that the problem was rawsz being zero and back tracking that took me some time.
Quoted 6 lines
With NULL
hash_algo, any dereferencing fails immediately with a nice stack trace. Another reason to push me towards NULL hash algo is, even if we prefer nice messages over segmentation faults, we can't avoid it completely anyway (empty_treeandempty_blobare still NULL in unknown hash algo and will cause segfaults). Might as well make things consistent and always segfault.
That makes sense. Thanks for the explanation.
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
v3 cover Fix initializing the_hash_algo
In reply to Nguyễn Thái Ngọc Duyv3 refines v2 a bit more:
- fix configure typo (and stray words in commit message)
- use repo_set_hash_algo() instead of reassigning the_hash_algo
- compare hash algos by format_id
- catch NULL hash algo, report nicely and suggest GIT_HASH_FIXUP
The last point makes me much happier about keeping this workaround around until we are confident we can live without it. Interdiff
builtin/index-pack.c +8 -5
@@ -331,21 +331,24 @@ static const char *open_pack_file(const char *pack_name)static void prepare_hash_algo (uint32_t pack_version) { +int pack_algo_id;const struct git_hash_algo *pack_algo;switch (pack_version) {case 2 :case 3 : - pack_algo = &hash_algos[GIT_HASH_SHA1 ]; + pack_algo_id =GIT_HASH_SHA1 ;break ;default : -die ("BUG: how to determine hash algo for new version?" ); +die ("BUG: how to determine hash algo for version %d?" , + pack_version); } -if (!the_hash_algo)/* running without repo */ - the_hash_algo = pack_algo; +if (!repo_has_valid_hash_algo (the_repository))/* running without repo */ +repo_set_hash_algo (the_repository, pack_algo_id); -if (the_hash_algo != pack_algo) + pack_algo = &hash_algos[pack_algo_id]; +if (the_hash_algo->format_id != pack_algo->format_id)die (_ ("incompatible hash algorithm, " "configured for %s but the pack file needs %s" ), the_hash_algo->name, pack_algo->name);
cache.h +1 -1
@@ -53,7 +53,7 @@ struct object_id {unsigned char hash[GIT_MAX_RAWSZ ]; }; -#define the_hash_algo the_repository->hash_algo +#define the_hash_algo repo_get_hash_algo(the_repository)#if defined(DT_UNKNOWN ) && !defined(NO_D_TYPE_IN_DIRENT )#define DTYPE (de) ((de)->d_type)
diff.c +3 -3
@@ -3997,15 +3997,15 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)/* * NEEDSWORK: When running in no-index mode (and no repo is -* found, thus no hash algo conifugred), fall back to SHA-1 +* found, thus no hash algo configured), fall back to SHA-1 * hashing (which is used by diff_fill_oid_info below) to * avoid regression in diff output. * * In future, perhaps we can allow the user to specify their * hash algorithm from command line in this mode. */ -if (o->flags.no_index && !the_hash_algo) - the_hash_algo = &hash_algos[GIT_HASH_SHA1 ]; +if (o->flags.no_index && !repo_has_valid_hash_algo (the_repository)) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 );diff_fill_oid_info (one);diff_fill_oid_info (two);
repository.h +16 -0
@@ -107,4 +107,20 @@ extern void repo_clear(struct repository *repo);*/ extern int repo_read_index (struct repository *repo); +static inline const struct git_hash_algo *repo_get_hash_algo ( +const struct repository *repo) +{ +if (!repo->hash_algo) +die ("BUG: hash_algo is not initialized!\n%s" , +_ ("You can work around this by setting environment" +" variable GIT_HASH_FIXUP=1.\n" +"Please report this to [email protected]" )); +return repo->hash_algo; +} + +static inline int repo_has_valid_hash_algo (const struct repository *repo) +{ +return repo->hash_algo !=NULL ; +} +#endif /* REPOSITORY_H */
Nguyễn Thái Ngọc Duy (6):
setup.c: initialize the_repository correctly in all cases sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN] cache.h: make the_hash_algo read-only
index-pack: check (and optionally set) hash algo based on input file diff.c: initialize hash algo when running in --no-index mode Revert "repository: pre-initialize hash algo pointer"
builtin/index-pack.c | 29 ++++++++++++++++++++++++++++- builtin/init-db.c | 3 ++- cache.h | 5 +++-- common-main.c | 10 ++++++++++ diff.c | 12 ++++++++++++ path.c | 2 +- repository.c | 2 +- repository.h | 16 ++++++++++++++++ setup.c | 5 ++++- sha1_file.c | 2 +- t/helper/test-dump-split-index.c | 2 ++ 11 files changed, 80 insertions(+), 8 deletions(-)
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH v3 1/6 setup.c: initialize the_repository correctly in all cases
There are many ways for any command to access a git repository:
- most of them will try to discover the .git dir via setup_git_directory() and friends
- the server side programs already know where the repo is and prepare with enter_repo()
- special commands that deal with repo creation (init/clone) use init_db() once the new repo is ready for access.
- somebody accesses $GIT_DIR before any of above functions are called and accidentally sets up a git repository by set_git_dir() alone
"the_repository" is partially set up via set_git_dir() at some point in all four cases. The hash algorithm though is configured later after .git/config is read.
So far proper repo initialization is done only for the first case [1]. The second case is not covered (but that's fine [3]). The third case was found and worked around in [2]. The fourth case is a buggy one, which should be fixed already by jk/no-looking-at-dotgit-outside-repo and never happens again.
This patch makes sure all cases initialize the hash algorithm in the_repository correctly. Both second and third cases must run check_repo_format() before "entering" it. Eventually we probably just rename this function to init_repo() or something.
[1] 78a6766802 (Integrate hash algorithm support with repo setup -
2017-11-12)
[2] e26f7f19b6 (repository: pre-initialize hash algo pointer -
2018-01-19)
[3] the reason server side is still running ok with no hash algo before
[2] is because the programs that use enter_repo() do very little (and unlikely to hash anything) then spawn a new program (like pack-objects or upload-archive) to do the heavy lifting. These programs already use setup_git_directory() or the gently version
--- builtin/init-db.c | 3 ++- cache.h | 3 ++- path.c | 2 +- setup.c | 5 ++++- 4 files changed, 9 insertions(+), 4 deletions(-)
builtin/init-db.c +2 -1
@@ -9,6 +9,7 @@#include "builtin.h" #include "exec_cmd.h" #include "parse-options.h" +#include "repository.h" #ifndef DEFAULT_GIT_TEMPLATE_DIR #define DEFAULT_GIT_TEMPLATE_DIR "/usr/share/git-core/templates"
@@ -369,7 +370,7 @@ int init_db(const char *git_dir, const char *real_git_dir,* config file, so this will not fail. What we are catching * is an attempt to reinitialize new repository with an old tool. */ -check_repository_format (); +check_repository_format (the_repository); reinit =create_default_files (template_dir, original_git_dir);
cache.h +2 -1
@@ -894,6 +894,7 @@ extern int repository_format_precious_objects;extern char *repository_format_partial_clone;extern const char *core_partial_clone_filter_default; +struct repository ;struct repository_format {int version;int precious_objects;
@@ -926,7 +927,7 @@ int verify_repository_format(const struct repository_format *format,* set_git_dir() before calling this, and use it only for "are we in a valid * repo?". */ -extern void check_repository_format (void ); +extern void check_repository_format (struct repository *);#define MTIME_CHANGED 0x0001#define CTIME_CHANGED 0x0002
path.c +1 -1
@@ -827,7 +827,7 @@ const char *enter_repo(const char *path, int strict)if (is_git_directory ("." )) {set_git_dir ("." ); -check_repository_format (); +check_repository_format (the_repository);return path; }
setup.c +4 -1
@@ -1180,11 +1180,14 @@ int git_config_perm(const char *var, const char *value)return -(i &0666 ); } -void check_repository_format (void ) +/* optionally configure "repo" to the correct format */ +void check_repository_format (struct repository *repo) {struct repository_format repo_fmt;check_repository_format_gently (get_git_dir (), &repo_fmt,NULL ); startup_info->have_repository =1 ; +if (repo) +repo_set_hash_algo (repo, repo_fmt.hash_algo); }/*
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH v3 2/6 sha1_file.c: keep a good name for "unknown" hash_algos[UNKNOWN]
In reply to Nguyễn Thái Ngọc DuyThis is mostly for displaying the hash algorithm name when we report errors. Printing "unknown" with '%s' is much better than '(null)' in glibc printf version (and probably could crash if other implementations do not check for NULL pointer)
--- sha1_file.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
sha1_file.c +1 -1
@@ -71,7 +71,7 @@ static void git_hash_unknown_final(unsigned char *hash, git_hash_ctx *ctx)const struct git_hash_algo hash_algos[GIT_HASH_NALGOS ] = { { -NULL , +"unknown" ,0x00000000 ,0 ,0 ,
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH v3 3/6 cache.h: make the_hash_algo read-only
In reply to Nguyễn Thái Ngọc DuyBy returning an R-value in the_hash_algo we make sure people can't accidentally change hash algorithm with
the_hash_algo = &hash_algos[something];
and go with repo_set_hash_algo() instead. Of course they can still do
the_repository->hash_algo = ...
but that is more obvious and easily caught in review.
--- cache.h | 2 +- repository.h | 6 ++++++ 2 files changed, 7 insertions(+), 1 deletion(-)
cache.h +1 -1
@@ -53,7 +53,7 @@ struct object_id {unsigned char hash[GIT_MAX_RAWSZ ]; }; -#define the_hash_algo the_repository->hash_algo +#define the_hash_algo repo_get_hash_algo(the_repository)#if defined(DT_UNKNOWN ) && !defined(NO_D_TYPE_IN_DIRENT )#define DTYPE (de) ((de)->d_type)
repository.h +6 -0
@@ -107,4 +107,10 @@ extern void repo_clear(struct repository *repo);*/ extern int repo_read_index (struct repository *repo); +static inline const struct git_hash_algo *repo_get_hash_algo ( +const struct repository *repo) +{ +return repo->hash_algo; +} +#endif /* REPOSITORY_H */
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH v3 4/6 index-pack: check (and optionally set) hash algo based on input file
In reply to Nguyễn Thái Ngọc DuyAfter 454253f059 (builtin/index-pack: improve hash function abstraction
- 2018-02-01), index-pack uses the_hash_algo for hashing. If "git index-pack" is executed without a repository, we do not know what hash algorithm to be used and the_hash_algo in theory could be undefined.
Since there should be some information about the hash algorithm in the input pack file, we can initialize the correct hash algorithm with that if the_hash_algo is not yet initialized. This assumes that pack files with new hash algorithm MUST step up pack version.
While at there, make sure the hash algorithm requested by the pack file and configured by the repository (if we're running with a repo) are consistent.
--- builtin/index-pack.c | 29 ++++++++++++++++++++++++++++- repository.h | 5 +++++ 2 files changed, 33 insertions(+), 1 deletion(-)
builtin/index-pack.c +28 -1
@@ -326,10 +326,34 @@ static const char *open_pack_file(const char *pack_name) output_fd =-1 ; nothread_data.pack_fd = input_fd; } - the_hash_algo->init_fn (&input_ctx);return pack_name; } +static void prepare_hash_algo (uint32_t pack_version) +{ +int pack_algo_id; +const struct git_hash_algo *pack_algo; + +switch (pack_version) { +case 2 : +case 3 : + pack_algo_id =GIT_HASH_SHA1 ; +break ; +default : +die ("BUG: how to determine hash algo for version %d?" , + pack_version); + } + +if (!repo_has_valid_hash_algo (the_repository))/* running without repo */ +repo_set_hash_algo (the_repository, pack_algo_id); + + pack_algo = &hash_algos[pack_algo_id]; +if (the_hash_algo->format_id != pack_algo->format_id) +die (_ ("incompatible hash algorithm, " +"configured for %s but the pack file needs %s" ), + the_hash_algo->name, pack_algo->name); +} +static void parse_pack_header (void ) {struct pack_header *hdr =fill (sizeof (struct pack_header ));
@@ -341,6 +365,9 @@ static void parse_pack_header(void)die (_ ("pack version %" PRIu32" unsupported" ),ntohl (hdr->hdr_version)); +prepare_hash_algo (ntohl (hdr->hdr_version)); + the_hash_algo->init_fn (&input_ctx); + nr_objects =ntohl (hdr->hdr_entries);use (sizeof (struct pack_header )); }
repository.h +5 -0
@@ -113,4 +113,9 @@ static inline const struct git_hash_algo *repo_get_hash_algo( return repo->hash_algo; } +static inline int repo_has_valid_hash_algo (const struct repository *repo) +{ +return repo->hash_algo !=NULL ; +} +#endif /* REPOSITORY_H */
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH v3 5/6 diff.c: initialize hash algo when running in --no-index mode
In reply to Nguyễn Thái Ngọc DuyOur "git diff" command supports running as a standalone tool. In this code path, we try to hash the file content but after 18e2588e11 (sha1_file: switch uses of SHA-1 to the_hash_algo - 2018-02-01), there is a chance that the_hash_algo (required by index_path) may still be uninitialized if no repository is found.
Executing index_path() when the_hash_algo is NULL (or points to unknown algo) either crashes or dies. Let's make it a bit safer by explicitly falling back to SHA-1 (so that the diff output remains the same as before, compared to the alternative that we simply do not hash).
--- diff.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
diff.c +12 -0
@@ -3995,6 +3995,18 @@ static void run_diff(struct diff_filepair *p, struct diff_options *o)return ; } +/* +* NEEDSWORK: When running in no-index mode (and no repo is +* found, thus no hash algo configured), fall back to SHA-1 +* hashing (which is used by diff_fill_oid_info below) to +* avoid regression in diff output. +* +* In future, perhaps we can allow the user to specify their +* hash algorithm from command line in this mode. +*/ +if (o->flags.no_index && !repo_has_valid_hash_algo (the_repository)) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +diff_fill_oid_info (one);diff_fill_oid_info (two);
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH v3 6/6 Revert "repository: pre-initialize hash algo pointer"
In reply to Nguyễn Thái Ngọc DuyThis reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root problem, git clone not setting up the_hash_algo, has been fixed in the previous patch.
Since this is a dangerous move and could potentially break stuff after release (and leads to workaround like the reverted commit), the workaround technically remains, but is hidden behind a new environment variable GIT_HASH_FIXUP. This should let the users continue to use git while we fix the problem. This variable can be deleted after one or two releases.
test-dump-split-index.c needs to call setup_git_directory() after this to configure hash algorithm before parsing the index file. It's not the best way (i.e. check index file version) but it's a test tool, this is good enough.
--- common-main.c | 10 ++++++++++ repository.c | 2 +- repository.h | 5 +++++ t/helper/test-dump-split-index.c | 2 ++ 4 files changed, 18 insertions(+), 1 deletion(-)
common-main.c +10 -0
@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the
@@ -40,5 +41,14 @@ int main(int argc, const char **argv)restore_sigpipe_to_default (); +/* +* Temporary recovery measure while hashing code is being +* refactored. This variable should be gone after everybody +* has used the_hash_algo in one or two releases and nobody +* complains anything. +*/ +if (getenv ("GIT_HASH_FIXUP" )) +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +return cmd_main (argc, argv); }
repository.c +1 -1
@@ -5,7 +5,7 @@/* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index,NULL ,0 ,0 };struct repository *the_repository = &the_repo;
repository.h +5 -0
@@ -110,6 +110,11 @@ extern int repo_read_index(struct repository *repo);static inline const struct git_hash_algo *repo_get_hash_algo (const struct repository *repo) { +if (!repo->hash_algo) +die ("BUG: hash_algo is not initialized!\n%s" , +_ ("You can work around this by setting environment" +" variable GIT_HASH_FIXUP=1.\n" +"Please report this to [email protected]" ));return repo->hash_algo; }
t/helper/test-dump-split-index.c +2 -0
@@ -12,6 +12,8 @@ int cmd_main(int ac, const char **av)struct split_index *si;int i; +setup_git_directory (); +do_read_index (&the_index, av[1 ],1 );printf ("own %s\n" ,sha1_to_hex (the_index.sha1)); si = the_index.split_index;
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH v3 0/6] Fix initializing the_hash_algo
In reply to Nguyễn Thái Ngọc DuyQuoted 9 lines · Nguyễn Thái Ngọc Duy
v3 refines v2 a bit more:
- fix configure typo (and stray words in commit message)
- userepo_set_hash_algo()instead of reassigningthe_hash_algo
- compare hash algos byformat_id
- catch NULL hash algo, report nicely and suggestGIT_HASH_FIXUPThe last point makes me much happier about keeping this workaround around until we are confident we can live without it. Interdiff
This looks sane to me.
Signature · 3 lines
brian m. carlson / brian with sandals: Houston, Texas, US https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204
Re: [PATCH v3 0/6] Fix initializing the_hash_algo
Quoted 8 lines · brian m. carlson
Quoted 7 lines · Nguyễn Thái Ngọc Duy
v3 refines v2 a bit more:
- fix configure typo (and stray words in commit message)
- userepo_set_hash_algo()instead of reassigningthe_hash_algo
- compare hash algos byformat_id
- catch NULL hash algo, report nicely and suggestGIT_HASH_FIXUP
Quoted 6 lines
Quoted 2 lines
The last point makes me much happier about keeping this workaround around until we are confident we can live without it. Interdiff
This looks sane to me.
Reviewed-by brian m. carlson <[email protected]>
I agree with this version as well.
Thanks,
Stefan
cover Delete ignore_env member in struct repository
In reply to Brandon WilliamsIt turns out I don't need my other series [1] in order to delete this field. This series moves getenv() calls from repo_set_gitdir()/repo_setup_env() and prepare_alt_odb() back in environment.c where they belong in my opinion.
The repo_set_gitdir() now takes $GIT_DIR and optionally all other configurable paths. If those paths are NULL, default repo layout will be used. With getenv() no longer called inside repo_set_gitdir(), ignore_env has no reason to stay. This is in 1/4.
The getenv() in prepare_alt_odb() is also moved back to setup_git_env() in 3/4. It demonstrates how we could move other getenv() back to if we want.
This series is built on top of Stefan's object-store-part1, v4. I could rebase it on 'master' too, but then Junio may need to resolve some conflicts.
[1] https://public-inbox.org/git/[email protected]/
Nguyễn Thái Ngọc Duy (4):
repository.c: move env-related setup code back to environment.c repository.c: delete dead functions
sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
repository: delete ignore_env member
cache.h | 2 +- environment.c | 13 +++++++-- object-store.h | 5 +++- object.c | 1 + repository.c | 79 ++++++++++++++++++++++---------------------------- repository.h | 21 +++++++------- setup.c | 3 +- sha1_file.c | 6 +--- 8 files changed, 64 insertions(+), 66 deletions(-)
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH 1/4 repository.c: move env-related setup code back to environment.c
It does not make sense that generic repository code contains handling of environment variables, which are specific for the main repository only. Refactor repo_set_gitdir() function to take $GIT_DIR and optionally _all_ other customizable paths. These optional paths can be NULL and will be calculated according to the default directory layout.
Note that some dead functions are left behind to reduce diff noise. They will be deleted in the next patch.
--- cache.h | 2 +- environment.c | 12 +++++++++--- repository.c | 48 ++++++++++++++++++++++++++++++++++++++++++------ repository.h | 11 ++++++++++- setup.c | 3 +-- 5 files changed, 63 insertions(+), 13 deletions(-)
cache.h +1 -1
@@ -459,7 +459,7 @@ static inline enum object_type object_type(unsigned int mode)*/ extern const char *const local_repo_env[]; -extern void setup_git_env (void ); +extern void setup_git_env (const char *git_dir);/* * Returns true iff we have a configured git repository (either via
environment.c +9 -3
@@ -148,10 +148,17 @@ static char *expand_namespace(const char *raw_namespace)return strbuf_detach (&buf,NULL ); } -void setup_git_env (void ) +void setup_git_env (const char *git_dir) {const char *shallow_file;const char *replace_ref_base; +struct set_gitdir_args args = {NULL }; + + args.shared_root =getenv (GIT_COMMON_DIR_ENVIRONMENT ); + args.object_dir =getenv (DB_ENVIRONMENT ); + args.graft_file =getenv (GRAFT_ENVIRONMENT ); + args.index_file =getenv (INDEX_ENVIRONMENT ); +repo_set_gitdir (the_repository, git_dir, &args);if (getenv (NO_REPLACE_OBJECTS_ENVIRONMENT )) check_replace_refs =0 ;
@@ -301,8 +308,7 @@ int set_git_dir(const char *path) {if (setenv (GIT_DIR_ENVIRONMENT , path,1 ))return error ("Could not set GIT_DIR to '%s'" , path); -repo_set_gitdir (the_repository, path); -setup_git_env (); +setup_git_env (path);return 0 ; }
repository.c +42 -6
70 lines not on this page · raw message
repository.h +10 -1
@@ -89,7 +89,16 @@ struct repository {extern struct repository *the_repository; -extern void repo_set_gitdir (struct repository *repo,const char *path); +struct set_gitdir_args { +const char *shared_root; +const char *object_dir; +const char *graft_file; +const char *index_file; +}; + +extern void repo_set_gitdir (struct repository *repo, +const char *root, +const struct set_gitdir_args *optional);extern void repo_set_worktree (struct repository *repo,const char *path);extern void repo_set_hash_algo (struct repository *repo,int algo);extern int repo_init (struct repository *repo,const char *gitdir,const char *worktree);
setup.c +1 -2
@@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)const char *gitdir =getenv (GIT_DIR_ENVIRONMENT );if (!gitdir) gitdir =DEFAULT_GIT_DIR_ENVIRONMENT ; -repo_set_gitdir (the_repository, gitdir); -setup_git_env (); +setup_git_env (gitdir); }if (startup_info->have_repository)repo_set_hash_algo (the_repository, repo_fmt.hash_algo);
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c
Quoted 14 lines · Nguyễn Thái Ngọc Duy
It does not make sense that generic repository code contains handling of environment variables, which are specific for the main repository only. Refactor
repo_set_gitdir()function to take $GIT_DIR and optionally _all_ other customizable paths. These optional paths can be NULL and will be calculated according to the default directory layout.Note that some dead functions are left behind to reduce diff noise. They will be deleted in the next patch.
Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>---
environment.c +0
-1@@ -148,10 +148,17 @@ static char *expand_namespace(const char *raw_namespace) -void setup_git_env (void )
environment.c static char *expand_namespace(const char *raw_namespace) · line 157
Quoted 10 lines
environment.c +7
-0+void setup_git_env (const char *git_dir) {const char *shallow_file;const char *replace_ref_base; +struct set_gitdir_args args = {NULL }; + + args.shared_root =getenv (GIT_COMMON_DIR_ENVIRONMENT ); + args.object_dir =getenv (DB_ENVIRONMENT ); + args.graft_file =getenv (GRAFT_ENVIRONMENT ); + args.index_file =getenv (INDEX_ENVIRONMENT );
According to POSIX[1], the result of getenv() may be invalidated by another call to getenv() (or setenv() or unsetenv() or putenv()).
[1]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/getenv.html
repository.c static void repo_setup_env(struct repository *repo) · line 69
Quoted 12 lines
environment.c +1
-0+repo_set_gitdir (the_repository, git_dir, &args);repository.c +8
-0@@ -61,15 +61,50 @@ static void repo_setup_env(struct repository *repo) +void repo_set_gitdir (struct repository *repo, +const char *root, +const struct set_gitdir_args *o) +{ +const char *gitfile =read_gitfile (root); +char *old_gitdir = repo->gitdir; + + repo->gitdir =xstrdup (gitfile ? gitfile : root);free (old_gitdir);
Can:
char *old_gitdir = repo->gitdir;
repo->gitdir = xstrdup(...);
free(old_gitdir);be simplified to:
free(repo->gitdir);
repo->gitdir = xstrdup(...);?
repository.c static void repo_setup_env(struct repository *repo) · line 101
Quoted 1 line
repository.c +1
-0+repo_set_commondir (repo, o->shared_root);
The repo_set_gitdir() prototype (below) makes it seem as if the last argument ('o', in this case) is optional, presumably by passing in NULL, however, this code does not appear to be prepared to deal with NULL.
Quoted 16 lines
repository.c +6
-0+expand_base_dir (&repo->objects.objectdir, o->object_dir, + repo->commondir,"objects" ); +expand_base_dir (&repo->graft_file, o->graft_file, + repo->commondir,"info/grafts" ); +expand_base_dir (&repo->index_file, o->index_file, + repo->gitdir,"index" ); }repository.h +7
-0@@ -89,7 +89,16 @@ struct repository { +struct set_gitdir_args { + [...] +}; + +extern void repo_set_gitdir (struct repository *repo, +const char *root, +const struct set_gitdir_args *optional);
Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c
Quoted 6 lines · Eric Sunshine
Quoted 1 line
+
repo_set_commondir(repo, o->shared_root);The
repo_set_gitdir()prototype (below) makes it seem as if the last argument ('o', in this case) is optional, presumably by passing in NULL, however, this code does not appear to be prepared to deal with NULL.
I know. I went this this struct because I really hate to write lots of NULL without names
repo_set_gitdir(gitdir, NULL, NULL, NULL, ...);
but I could not find a nicer way to make the whole struct optional, not without repeating more code or using preprocessor. Given that this function has 3-4 call sites max, I think it's still ok.
Quoted 8 lines
Quoted 8 lines
repository.c +6
-0+expand_base_dir (&repo->objects.objectdir, o->object_dir, + repo->commondir,"objects" ); +expand_base_dir (&repo->graft_file, o->graft_file, + repo->commondir,"info/grafts" ); +expand_base_dir (&repo->index_file, o->index_file, + repo->gitdir,"index" ); }
Signature · 1 line
Duy
Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c
In reply to Eric SunshineQuoted 6 lines · Eric Sunshine
Can:
char *old_gitdir = repo->gitdir; repo->gitdir = xstrdup(...); free(old_gitdir);
Quoted 6 lines
be simplified to:
free(repo->gitdir); repo->gitdir = xstrdup(...);?
No because I think "root" (not quoted here) could point to the same value as repo->gitdir, if you free that first, the second xstrdup() will duplicate a freed memory. See 1fb2b636c6 (set_git_dir: handle feeding gitdir to itself - 2017-09-05)
I will add a comment about this since it's not very clear from this code.
Signature · 1 line
Duy
Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c
In reply to Nguyễn Thái Ngọc Duysetup.c const char *setup_git_directory_gently(int *nongit_ok) · line 1120
Quoted 12 lines
setup.c +1
-2@@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)const char *gitdir =getenv (GIT_DIR_ENVIRONMENT );if (!gitdir) gitdir =DEFAULT_GIT_DIR_ENVIRONMENT ; -repo_set_gitdir (the_repository, gitdir); -setup_git_env (); +setup_git_env (gitdir); }
What makes git_dir special, such that we have to pass it in here? Could the check for getenv(GIT_DIR_ENVIRONMENT) and fallback to DEFAULT_... be part of setup_git_env() ?
Oh I guess that cannot be done easily as the submodule code spcifically doesn't want to discover the git dir itself.
Quoted 5 lines
setup.c +0
-1if (startup_info->have_repository)repo_set_hash_algo (the_repository, repo_fmt.hash_algo); --2.16.1.435.g8f24da2e1a
Re: [PATCH 1/4] repository.c: move env-related setup code back to environment.c
Quoted 10 lines · Stefan Beller
Quoted 10 lines
setup.c +0
-2@@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)const char *gitdir =getenv (GIT_DIR_ENVIRONMENT );if (!gitdir) gitdir =DEFAULT_GIT_DIR_ENVIRONMENT ; -repo_set_gitdir (the_repository, gitdir); -setup_git_env ();
Quoted 8 lines
setup.c const char *setup_git_directory_gently(int *nongit_ok) · line 1120
Quoted 2 lines
setup.c +1
-0+setup_git_env (gitdir); }What makes
git_dirspecial, such that we have to pass it in here? Could the check forgetenv(GIT_DIR_ENVIRONMENT)and fallback toDEFAULT_... be part ofsetup_git_env()?
Oh I guess that cannot be done easily as the submodule code spcifically doesn't want to discover the git dir itself.
No, submodule code already does not care about setup_git_env().
Back to your first question, this is related to the "NEEDSWORK" comment in this code. We _should_ always set_git_dir() before leaving setup_git_directory_gently() then everybody behaves the same way and we don't need this special setup_git_env() here at all. BUT we need to be careful about setting environment variable $GIT_DIR, done inside set_git_dir(), because sub-processes will inherit it and if we're not careful, we'll break .git discovery in those. There's another thing I don't like about this is setup_work_tree() using set_git_dir() the second time to adjust relative $GIT_DIR and friends when it moves $CWD. I'll need to fix this, soon.
So in short, it's special because the current situation is messy and (hopefully) temporary. But it should eventually be gone.
Signature · 1 line
Duy
PATCH 2/4 repository.c: delete dead functions
In reply to Nguyễn Thái Ngọc Duy--- repository.c | 45 --------------------------------------------- 1 file changed, 45 deletions(-)
repository.c +0 -45
@@ -16,51 +16,6 @@ static struct repository the_repo = { };struct repository *the_repository = &the_repo; -static char *git_path_from_env (const char *envvar,const char *git_dir, -const char *path,int fromenv) -{ -if (fromenv) { -const char *value =getenv (envvar); -if (value) -return xstrdup (value); - } - -return xstrfmt ("%s/%s" , git_dir, path); -} - -static int find_common_dir (struct strbuf *sb,const char *gitdir,int fromenv) -{ -if (fromenv) { -const char *value =getenv (GIT_COMMON_DIR_ENVIRONMENT ); -if (value) { -strbuf_addstr (sb, value); -return 1 ; - } - } - -return get_common_dir_noenv (sb, gitdir); -} - -static void repo_setup_env (struct repository *repo) -{ -struct strbuf sb =STRBUF_INIT ; - - repo->different_commondir =find_common_dir (&sb, repo->gitdir, - !repo->ignore_env); -free (repo->commondir); - repo->commondir =strbuf_detach (&sb,NULL ); -raw_object_store_clear (&repo->objects); - repo->objects.objectdir = -git_path_from_env (DB_ENVIRONMENT , repo->commondir, -"objects" , !repo->ignore_env); -free (repo->graft_file); - repo->graft_file =git_path_from_env (GRAFT_ENVIRONMENT , repo->commondir, -"info/grafts" , !repo->ignore_env); -free (repo->index_file); - repo->index_file =git_path_from_env (INDEX_ENVIRONMENT , repo->gitdir, -"index" , !repo->ignore_env); -} -static void expand_base_dir (char **out,const char *in,const char *base_dir,const char *def_in) {
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH 3/4 sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
In reply to Nguyễn Thái Ngọc Duygetenv() is supposed to work on the main repository only. This delayed getenv() code in sha1_file.c makes it more difficult to convert sha1_file.c to a generic object store that could be used by both submodule and main repositories.
Move the getenv() back in setup_git_env() where other env vars are also fetched.
--- environment.c | 1 + object-store.h | 5 ++++- object.c | 1 + repository.c | 2 ++ repository.h | 1 + sha1_file.c | 6 +----- 6 files changed, 10 insertions(+), 6 deletions(-)
environment.c +1 -0
@@ -158,6 +158,7 @@ void setup_git_env(const char *git_dir) args.object_dir =getenv (DB_ENVIRONMENT ); args.graft_file =getenv (GRAFT_ENVIRONMENT ); args.index_file =getenv (INDEX_ENVIRONMENT ); + args.alternate_db =getenv (ALTERNATE_DB_ENVIRONMENT );repo_set_gitdir (the_repository, git_dir, &args);if (getenv (NO_REPLACE_OBJECTS_ENVIRONMENT ))
object-store.h +4 -1
@@ -87,6 +87,9 @@ struct raw_object_store {*/ char *objectdir; +/* Path to extra alternate object database if not NULL */ +char *alternate_db; +struct packed_git *packed_git;/* A most-recently-used ordered version of the packed_git list. */ struct list_head packed_git_mru;
@@ -109,7 +112,7 @@ struct raw_object_store { unsigned packed_git_initialized : 1; }; -#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 } +#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);
object.c +1 -0
@@ -464,6 +464,7 @@ static void free_alt_odbs(struct raw_object_store *o)void raw_object_store_clear (struct raw_object_store *o) {FREE_AND_NULL (o->objectdir); +FREE_AND_NULL (o->alternate_db);free_alt_odbs (o); o->alt_odb_tail =NULL ;
repository.c +2 -0
@@ -56,6 +56,8 @@ void repo_set_gitdir(struct repository *repo,repo_set_commondir (repo, o->shared_root);expand_base_dir (&repo->objects.objectdir, o->object_dir, repo->commondir,"objects" ); +free (repo->objects.alternate_db); + repo->objects.alternate_db =xstrdup_or_null (o->alternate_db);expand_base_dir (&repo->graft_file, o->graft_file, repo->commondir,"info/grafts" ); expand_base_dir(&repo->index_file, o->index_file,
repository.h +1 -0
@@ -94,6 +94,7 @@ struct set_gitdir_args {const char *object_dir;const char *graft_file;const char *index_file; +const char *alternate_db; }; extern void repo_set_gitdir(struct repository *repo,
sha1_file.c +1 -5
@@ -673,15 +673,11 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)void prepare_alt_odb (struct repository *r) { -const char *alt; -if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - r->objects.alt_odb_tail = &r->objects.alt_odb_list; -link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (r, r->objects.alternate_db,PATH_SEP ,NULL ,0 );read_info_alternates (r, r->objects.objectdir,0 ); }
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH 4/4 repository: delete ignore_env member
In reply to Nguyễn Thái Ngọc DuyThis variable was added because the repo_set_gitdir() was created to cover both submodule and main repos, but these two are initialized a bit differently so ignore_env == 0 means main repo, while ignore_env != 0 is submodules.
Since the difference part (env variables) has been moved out of repo_set_gitdir(), this function works the same way for both repo types and ignore_env is not needed anymore.
--- repository.c | 4 +--- repository.h | 9 --------- 2 files changed, 1 insertion(+), 12 deletions(-)
repository.c +1 -3
@@ -12,7 +12,7 @@ static struct repository the_repo = {NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ], - 0, 0 + 0 };struct repository *the_repository = &the_repo;
@@ -134,8 +134,6 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)struct repository_format format;memset (repo,0 ,sizeof (*repo)); - repo->ignore_env =1 ; -INIT_LIST_HEAD (&repo->objects.packed_git_mru);if (repo_init_gitdir (repo, gitdir))
repository.h +0 -9
@@ -73,15 +73,6 @@ struct repository {const struct git_hash_algo *hash_algo;/* Configurations */ -/* -* Bit used during initialization to indicate if repository state (like -* the location of the 'objectdir') should be read from the -* environment. By default this bit will be set at the begining of -* 'repo_init()' so that all repositories will ignore the environment. -* The exception to this is 'the_repository', which doesn't go through -* the normal 'repo_init()' process. -*/ - unsigned ignore_env:1;/* Indicate if a repository has a different 'commondir' from 'gitdir' */ unsigned different_commondir:1;
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH 0/4] Delete ignore_env member in struct repository
In reply to Nguyễn Thái Ngọc DuyQuoted 3 lines · Nguyễn Thái Ngọc Duy
This series is built on top of Stefan's object-store-part1, v4. I could rebase it on 'master' too, but then Junio may need to resolve some conflicts.
As a follow-up fix for combined "move things to the_repo as much as possible" efforts I think what you did makes most sense. I would say that it makes even more sense to make these part of the object-store series if/when the series needs rerolling--after all the topic is already a multi-author effort.
I'll have to read them carefully before commenting on the actual patches ;-) Thanks.
Quoted 18 lines
[1] https://public-inbox.org/git/[email protected]/
Nguyễn Thái Ngọc Duy (4):
repository.c: move env-related setup code back toenvironment.crepository.c: delete dead functions
sha1_file.c: move delayed getenv(altdb) back tosetup_git_env()
repository: deleteignore_envmembercache.h | 2 +- environment.c | 13 +++++++-- object-store.h | 5 +++- object.c | 1 + repository.c | 79 ++++++++++++++++++++++---------------------------- repository.h | 21 +++++++------- setup.c | 3 +- sha1_file.c | 6 +--- 8 files changed, 64 insertions(+), 66 deletions(-)
Re: [PATCH 0/4] Delete ignore_env member in struct repository
In reply to Nguyễn Thái Ngọc DuyQuoted 10 lines · Nguyễn Thái Ngọc Duy
It turns out I don't need my other series [1] in order to delete this field. This series moves
getenv()calls from repo_set_gitdir()/repo_setup_env() andprepare_alt_odb()back inenvironment.cwhere they belong in my opinion.The
repo_set_gitdir()now takes $GIT_DIR and optionally all other configurable paths. If those paths are NULL, default repo layout will be used. Withgetenv()no longer called insiderepo_set_gitdir(),ignore_envhas no reason to stay. This is in 1/4.
Quoted 8 lines
The
getenv()inprepare_alt_odb()is also moved back tosetup_git_env()in 3/4. It demonstrates how we could move othergetenv()back to if we want.This series is built on top of Stefan's object-store-part1, v4. I could rebase it on 'master' too, but then Junio may need to resolve some conflicts.
Thanks for working on this,
I found this series a pleasant read, the only issue I saw was Erics upbringing of multiple getenv calls without strdup()ing the content.
What is the plan from here on?
Should I build further series on top of yours? The next series will focus on the pack side of things (pack.h, packfile.{c,h})
So maybe we'll have Junio merge down my series (and yours as it needs one reroll?) and then build on top of that?
Thanks,
Stefan
Re: [PATCH 0/4] Delete ignore_env member in struct repository
Quoted 6 lines · Stefan Beller
What is the plan from here on?
Should I build further series on top of yours? The next series will focus on the pack side of things (pack.h, packfile.{c,h})So maybe we'll have Junio merge down my series (and yours as it needs one reroll?) and then build on top of that?
I think that's the less painful way for everybody.
Signature · 1 line
Duy
v2 cover Delete ignore_env member in struct repository
In reply to Nguyễn Thái Ngọc Duyv2 fixes the incorrect use of consecutive getenv() and adds a comment to clarify the role of old_gitdir
Interdiff:
environment.c +22 -5
@@ -14,6 +14,7 @@#include "fmt-merge-msg.h" #include "commit.h" #include "object-store.h" +#include "argv-array.h" int trust_executable_bit =1 ;int trust_ctime =1 ;
@@ -148,18 +149,34 @@ static char *expand_namespace(const char *raw_namespace)return strbuf_detach (&buf,NULL ); } +/* Wrapper of getenv() that returns a strdup value. This value is kept +* in argv to be freed later. +*/ +static const char *getenv_safe (struct argv_array *argv,const char *name) +{ +const char *value =getenv (name); + +if (!value) +return NULL ; + +argv_array_push (argv, value); +return argv->argv[argv->argc -1 ]; +} +void setup_git_env (const char *git_dir) {const char *shallow_file;const char *replace_ref_base;struct set_gitdir_args args = {NULL }; +struct argv_array to_free =ARGV_ARRAY_INIT ; - args.shared_root =getenv (GIT_COMMON_DIR_ENVIRONMENT ); - args.object_dir =getenv (DB_ENVIRONMENT ); - args.graft_file =getenv (GRAFT_ENVIRONMENT ); - args.index_file =getenv (INDEX_ENVIRONMENT ); - args.alternate_db =getenv (ALTERNATE_DB_ENVIRONMENT ); + args.shared_root =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); + args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); + args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); + args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT ); + args.alternate_db =getenv_safe (&to_free,ALTERNATE_DB_ENVIRONMENT );repo_set_gitdir (the_repository, git_dir, &args); +argv_array_clear (&to_free);if (getenv (NO_REPLACE_OBJECTS_ENVIRONMENT )) check_replace_refs =0 ;
repository.c +5 -0
@@ -48,6 +48,11 @@ void repo_set_gitdir(struct repository *repo, const struct set_gitdir_args *o) {const char *gitfile =read_gitfile (root); +/* +* repo->gitdir is saved because the caller could pass "root" +* that also points to repo->gitdir. We want to keep it alive +* until after xstrdup(root). Then we can free it. +*/ char *old_gitdir = repo->gitdir; repo->gitdir =xstrdup (gitfile ? gitfile : root);
Nguyễn Thái Ngọc Duy (4):
repository.c: move env-related setup code back to environment.c repository.c: delete dead functions
sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
repository: delete ignore_env member
cache.h | 2 +- environment.c | 30 ++++++++++++++++-- object-store.h | 5 ++- object.c | 1 + repository.c | 84 ++++++++++++++++++++++++-------------------------- repository.h | 21 +++++++------ setup.c | 3 +- sha1_file.c | 6 +--- 8 files changed, 86 insertions(+), 66 deletions(-)
Signature · 1 line
2.16.1.435.g8f24da2e1a
PATCH v2 1/4 repository.c: move env-related setup code back to environment.c
It does not make sense that generic repository code contains handling of environment variables, which are specific for the main repository only. Refactor repo_set_gitdir() function to take $GIT_DIR and optionally _all_ other customizable paths. These optional paths can be NULL and will be calculated according to the default directory layout.
Note that some dead functions are left behind to reduce diff noise. They will be deleted in the next patch.
--- cache.h | 2 +- environment.c | 29 +++++++++++++++++++++++++--- repository.c | 53 +++++++++++++++++++++++++++++++++++++++++++++------ repository.h | 11 ++++++++++- setup.c | 3 +-- 5 files changed, 85 insertions(+), 13 deletions(-)
cache.h +1 -1
@@ -459,7 +459,7 @@ static inline enum object_type object_type(unsigned int mode)*/ extern const char *const local_repo_env[]; -extern void setup_git_env (void ); +extern void setup_git_env (const char *git_dir);/* * Returns true iff we have a configured git repository (either via
environment.c +26 -3
@@ -14,6 +14,7 @@#include "fmt-merge-msg.h" #include "commit.h" #include "object-store.h" +#include "argv-array.h" int trust_executable_bit =1 ;int trust_ctime =1 ;
@@ -148,10 +149,33 @@ static char *expand_namespace(const char *raw_namespace)return strbuf_detach (&buf,NULL ); } -void setup_git_env (void ) +/* Wrapper of getenv() that returns a strdup value. This value is kept +* in argv to be freed later. +*/ +static const char *getenv_safe (struct argv_array *argv,const char *name) +{ +const char *value =getenv (name); + +if (!value) +return NULL ; + +argv_array_push (argv, value); +return argv->argv[argv->argc -1 ]; +} + +void setup_git_env (const char *git_dir) {const char *shallow_file;const char *replace_ref_base; +struct set_gitdir_args args = {NULL }; +struct argv_array to_free =ARGV_ARRAY_INIT ; + + args.shared_root =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); + args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); + args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); + args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT ); +repo_set_gitdir (the_repository, git_dir, &args); +argv_array_clear (&to_free);if (getenv (NO_REPLACE_OBJECTS_ENVIRONMENT )) check_replace_refs =0 ;
@@ -301,8 +325,7 @@ int set_git_dir(const char *path) {if (setenv (GIT_DIR_ENVIRONMENT , path,1 ))return error ("Could not set GIT_DIR to '%s'" , path); -repo_set_gitdir (the_repository, path); -setup_git_env (); +setup_git_env (path);return 0 ; }
repository.c +47 -6
75 lines not on this page · raw message
repository.h +10 -1
@@ -89,7 +89,16 @@ struct repository {extern struct repository *the_repository; -extern void repo_set_gitdir (struct repository *repo,const char *path); +struct set_gitdir_args { +const char *shared_root; +const char *object_dir; +const char *graft_file; +const char *index_file; +}; + +extern void repo_set_gitdir (struct repository *repo, +const char *root, +const struct set_gitdir_args *optional);extern void repo_set_worktree (struct repository *repo,const char *path);extern void repo_set_hash_algo (struct repository *repo,int algo);extern int repo_init (struct repository *repo,const char *gitdir,const char *worktree);
setup.c +1 -2
@@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)const char *gitdir =getenv (GIT_DIR_ENVIRONMENT );if (!gitdir) gitdir =DEFAULT_GIT_DIR_ENVIRONMENT ; -repo_set_gitdir (the_repository, gitdir); -setup_git_env (); +setup_git_env (gitdir); }if (startup_info->have_repository)repo_set_hash_algo (the_repository, repo_fmt.hash_algo);
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH v2 1/4] repository.c: move env-related setup code back to environment.c
Quoted 175 lines · Nguyễn Thái Ngọc Duy
It does not make sense that generic repository code contains handling of environment variables, which are specific for the main repository only. Refactor
repo_set_gitdir()function to take $GIT_DIR and optionally _all_ other customizable paths. These optional paths can be NULL and will be calculated according to the default directory layout.Note that some dead functions are left behind to reduce diff noise. They will be deleted in the next patch.
Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>--- cache.h | 2 +- environment.c | 29 +++++++++++++++++++++++++--- repository.c | 53 +++++++++++++++++++++++++++++++++++++++++++++------ repository.h | 11 ++++++++++- setup.c | 3 +-- 5 files changed, 85 insertions(+), 13 deletions(-)cache.h +1
-1@@ -459,7 +459,7 @@ static inline enum object_type object_type(unsigned int mode)*/ extern const char *const local_repo_env[]; -extern void setup_git_env (void ); +extern void setup_git_env (const char *git_dir);/* * Returns true iff we have a configured git repository (either via environment.c +26
-3@@ -14,6 +14,7 @@#include "fmt-merge-msg.h" #include "commit.h" #include "object-store.h" +#include "argv-array.h" int trust_executable_bit =1 ;int trust_ctime =1 ;@@ -148,10 +149,33 @@ static char *expand_namespace(const char *raw_namespace)return strbuf_detach (&buf,NULL ); } -void setup_git_env (void ) +/* Wrapper of getenv() that returns a strdup value. This value is kept +* in argv to be freed later. +*/ +static const char *getenv_safe (struct argv_array *argv,const char *name) +{ +const char *value =getenv (name); + +if (!value) +return NULL ; + +argv_array_push (argv, value); +return argv->argv[argv->argc -1 ]; +} + +void setup_git_env (const char *git_dir) {const char *shallow_file;const char *replace_ref_base; +struct set_gitdir_args args = {NULL }; +struct argv_array to_free =ARGV_ARRAY_INIT ; + + args.shared_root =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); + args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); + args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); + args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT ); +repo_set_gitdir (the_repository, git_dir, &args); +argv_array_clear (&to_free);if (getenv (NO_REPLACE_OBJECTS_ENVIRONMENT )) check_replace_refs =0 ;@@ -301,8 +325,7 @@ int set_git_dir(const char *path) {if (setenv (GIT_DIR_ENVIRONMENT , path,1 ))return error ("Could not set GIT_DIR to '%s'" , path); -repo_set_gitdir (the_repository, path); -setup_git_env (); +setup_git_env (path);return 0 ; }repository.c +47
-6@@ -61,15 +61,55 @@ static void repo_setup_env(struct repository *repo) "index", !repo->ignore_env); } -void repo_set_gitdir (struct repository *repo,const char *path) +static void expand_base_dir (char **out,const char *in, +const char *base_dir,const char *def_in) { -const char *gitfile =read_gitfile (path); -char *old_gitdir = repo->gitdir; +free (*out); +if (in) + *out =xstrdup (in); +else + *out =xstrfmt ("%s/%s" , base_dir, def_in); +} + +static void repo_set_commondir (struct repository *repo, +const char *shared_root) +{ +struct strbuf sb =STRBUF_INIT ; + +free (repo->commondir); - repo->gitdir =xstrdup (gitfile ? gitfile : path); -repo_setup_env (repo); +if (shared_root) { + repo->different_commondir =1 ; + repo->commondir =xstrdup (shared_root); +return ; + } + repo->different_commondir =get_common_dir_noenv (&sb, repo->gitdir); + repo->commondir =strbuf_detach (&sb,NULL ); +} + +void repo_set_gitdir (struct repository *repo, +const char *root, +const struct set_gitdir_args *o) +{ +const char *gitfile =read_gitfile (root); +/* +* repo->gitdir is saved because the caller could pass "root" +* that also points to repo->gitdir. We want to keep it alive +* until after xstrdup(root). Then we can free it. +*/ +char *old_gitdir = repo->gitdir; + + repo->gitdir =xstrdup (gitfile ? gitfile : root);free (old_gitdir); + +repo_set_commondir (repo, o->shared_root); +expand_base_dir (&repo->objects.objectdir, o->object_dir, + repo->commondir,"objects" ); +expand_base_dir (&repo->graft_file, o->graft_file, + repo->commondir,"info/grafts" ); +expand_base_dir (&repo->index_file, o->index_file, + repo->gitdir,"index" ); }void repo_set_hash_algo (struct repository *repo,int hash_algo)@@ -87,6 +127,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir)int error =0 ;char *abspath =NULL ;const char *resolved_gitdir; +struct set_gitdir_args args = {NULL }; abspath =real_pathdup (gitdir,0 );if (!abspath) {@@ -101,7 +142,7 @@ static int repo_init_gitdir(struct repository *repo, const char *gitdir) goto out; } -repo_set_gitdir (repo, resolved_gitdir); +repo_set_gitdir (repo, resolved_gitdir, &args); out:free (abspath);repository.h +0
-0
repository.h struct repository { · line 93
Quoted 6 lines
repository.h +2
-1@@ -89,7 +89,16 @@ struct repository {extern struct repository *the_repository; -extern void repo_set_gitdir (struct repository *repo,const char *path); +struct set_gitdir_args { +const char *shared_root;
Can you add a comment explaining what shared_root is? From reading the code it seems to be the common gitdir but its not clear from just reading this.
Quoted 28 lines
repository.h +8
-0+const char *object_dir; +const char *graft_file; +const char *index_file; +}; + +extern void repo_set_gitdir (struct repository *repo, +const char *root, +const struct set_gitdir_args *optional);extern void repo_set_worktree (struct repository *repo,const char *path);extern void repo_set_hash_algo (struct repository *repo,int algo);extern int repo_init (struct repository *repo,const char *gitdir,const char *worktree);setup.c +1
-2@@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)const char *gitdir =getenv (GIT_DIR_ENVIRONMENT );if (!gitdir) gitdir =DEFAULT_GIT_DIR_ENVIRONMENT ; -repo_set_gitdir (the_repository, gitdir); -setup_git_env (); +setup_git_env (gitdir); }if (startup_info->have_repository)repo_set_hash_algo (the_repository, repo_fmt.hash_algo);Signature · 1 line
2.16.1.435.g8f24da2e1a
Signature · 1 line
Brandon Williams
PATCH v2 2/4 repository.c: delete dead functions
In reply to Nguyễn Thái Ngọc Duy--- repository.c | 45 --------------------------------------------- 1 file changed, 45 deletions(-)
repository.c +0 -45
@@ -16,51 +16,6 @@ static struct repository the_repo = { };struct repository *the_repository = &the_repo; -static char *git_path_from_env (const char *envvar,const char *git_dir, -const char *path,int fromenv) -{ -if (fromenv) { -const char *value =getenv (envvar); -if (value) -return xstrdup (value); - } - -return xstrfmt ("%s/%s" , git_dir, path); -} - -static int find_common_dir (struct strbuf *sb,const char *gitdir,int fromenv) -{ -if (fromenv) { -const char *value =getenv (GIT_COMMON_DIR_ENVIRONMENT ); -if (value) { -strbuf_addstr (sb, value); -return 1 ; - } - } - -return get_common_dir_noenv (sb, gitdir); -} - -static void repo_setup_env (struct repository *repo) -{ -struct strbuf sb =STRBUF_INIT ; - - repo->different_commondir =find_common_dir (&sb, repo->gitdir, - !repo->ignore_env); -free (repo->commondir); - repo->commondir =strbuf_detach (&sb,NULL ); -raw_object_store_clear (&repo->objects); - repo->objects.objectdir = -git_path_from_env (DB_ENVIRONMENT , repo->commondir, -"objects" , !repo->ignore_env); -free (repo->graft_file); - repo->graft_file =git_path_from_env (GRAFT_ENVIRONMENT , repo->commondir, -"info/grafts" , !repo->ignore_env); -free (repo->index_file); - repo->index_file =git_path_from_env (INDEX_ENVIRONMENT , repo->gitdir, -"index" , !repo->ignore_env); -} -static void expand_base_dir (char **out,const char *in,const char *base_dir,const char *def_in) {
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH v2 2/4] repository.c: delete dead functions
Quoted 1 line · Nguyễn Thái Ngọc Duy
Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>
These could technically be removed in the previous patch, it may make understanding what is happening in the previous patch a bit clearer too.
Quoted 63 lines
--- repository.c | 45 --------------------------------------------- 1 file changed, 45 deletions(-)repository.c +0
-45@@ -16,51 +16,6 @@ static struct repository the_repo = { };struct repository *the_repository = &the_repo; -static char *git_path_from_env (const char *envvar,const char *git_dir, -const char *path,int fromenv) -{ -if (fromenv) { -const char *value =getenv (envvar); -if (value) -return xstrdup (value); - } - -return xstrfmt ("%s/%s" , git_dir, path); -} - -static int find_common_dir (struct strbuf *sb,const char *gitdir,int fromenv) -{ -if (fromenv) { -const char *value =getenv (GIT_COMMON_DIR_ENVIRONMENT ); -if (value) { -strbuf_addstr (sb, value); -return 1 ; - } - } - -return get_common_dir_noenv (sb, gitdir); -} - -static void repo_setup_env (struct repository *repo) -{ -struct strbuf sb =STRBUF_INIT ; - - repo->different_commondir =find_common_dir (&sb, repo->gitdir, - !repo->ignore_env); -free (repo->commondir); - repo->commondir =strbuf_detach (&sb,NULL ); -raw_object_store_clear (&repo->objects); - repo->objects.objectdir = -git_path_from_env (DB_ENVIRONMENT , repo->commondir, -"objects" , !repo->ignore_env); -free (repo->graft_file); - repo->graft_file =git_path_from_env (GRAFT_ENVIRONMENT , repo->commondir, -"info/grafts" , !repo->ignore_env); -free (repo->index_file); - repo->index_file =git_path_from_env (INDEX_ENVIRONMENT , repo->gitdir, -"index" , !repo->ignore_env); -} -static void expand_base_dir (char **out,const char *in,const char *base_dir,const char *def_in) {Signature · 1 line
2.16.1.435.g8f24da2e1a
Signature · 1 line
Brandon Williams
PATCH v2 3/4 sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
In reply to Nguyễn Thái Ngọc Duygetenv() is supposed to work on the main repository only. This delayed getenv() code in sha1_file.c makes it more difficult to convert sha1_file.c to a generic object store that could be used by both submodule and main repositories.
Move the getenv() back in setup_git_env() where other env vars are also fetched.
--- environment.c | 1 + object-store.h | 5 ++++- object.c | 1 + repository.c | 2 ++ repository.h | 1 + sha1_file.c | 6 +----- 6 files changed, 10 insertions(+), 6 deletions(-)
environment.c +1 -0
@@ -174,6 +174,7 @@ void setup_git_env(const char *git_dir) args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT ); + args.alternate_db =getenv_safe (&to_free,ALTERNATE_DB_ENVIRONMENT );repo_set_gitdir (the_repository, git_dir, &args);argv_array_clear (&to_free);
object-store.h +4 -1
@@ -87,6 +87,9 @@ struct raw_object_store {*/ char *objectdir; +/* Path to extra alternate object database if not NULL */ +char *alternate_db; +struct packed_git *packed_git;/* A most-recently-used ordered version of the packed_git list. */ struct list_head packed_git_mru;
@@ -109,7 +112,7 @@ struct raw_object_store { unsigned packed_git_initialized : 1; }; -#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 } +#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);
object.c +1 -0
@@ -464,6 +464,7 @@ static void free_alt_odbs(struct raw_object_store *o)void raw_object_store_clear (struct raw_object_store *o) {FREE_AND_NULL (o->objectdir); +FREE_AND_NULL (o->alternate_db);free_alt_odbs (o); o->alt_odb_tail =NULL ;
repository.c +2 -0
@@ -61,6 +61,8 @@ void repo_set_gitdir(struct repository *repo,repo_set_commondir (repo, o->shared_root);expand_base_dir (&repo->objects.objectdir, o->object_dir, repo->commondir,"objects" ); +free (repo->objects.alternate_db); + repo->objects.alternate_db =xstrdup_or_null (o->alternate_db);expand_base_dir (&repo->graft_file, o->graft_file, repo->commondir,"info/grafts" ); expand_base_dir(&repo->index_file, o->index_file,
repository.h +1 -0
@@ -94,6 +94,7 @@ struct set_gitdir_args {const char *object_dir;const char *graft_file;const char *index_file; +const char *alternate_db; }; extern void repo_set_gitdir(struct repository *repo,
sha1_file.c +1 -5
@@ -673,15 +673,11 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)void prepare_alt_odb (struct repository *r) { -const char *alt; -if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - r->objects.alt_odb_tail = &r->objects.alt_odb_list; -link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (r, r->objects.alternate_db,PATH_SEP ,NULL ,0 );read_info_alternates (r, r->objects.objectdir,0 ); }
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH v2 3/4] sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
Quoted 70 lines · Nguyễn Thái Ngọc Duy
getenv()is supposed to work on the main repository only. This delayedgetenv()code insha1_file.cmakes it more difficult to convertsha1_file.cto a generic object store that could be used by both submodule and main repositories.Move the
getenv()back insetup_git_env()where other env vars are also fetched.Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>--- environment.c | 1 + object-store.h | 5 ++++- object.c | 1 + repository.c | 2 ++ repository.h | 1 + sha1_file.c | 6 +----- 6 files changed, 10 insertions(+), 6 deletions(-)environment.c +1
-0@@ -174,6 +174,7 @@ void setup_git_env(const char *git_dir) args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT ); + args.alternate_db =getenv_safe (&to_free,ALTERNATE_DB_ENVIRONMENT );repo_set_gitdir (the_repository, git_dir, &args);argv_array_clear (&to_free);object-store.h +4
-1@@ -87,6 +87,9 @@ struct raw_object_store {*/ char *objectdir; +/* Path to extra alternate object database if not NULL */ +char *alternate_db; +struct packed_git *packed_git;/* A most-recently-used ordered version of the packed_git list. */ struct list_head packed_git_mru;@@ -109,7 +112,7 @@ struct raw_object_store { unsigned packed_git_initialized : 1; }; -#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 } +#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);object.c +1
-0@@ -464,6 +464,7 @@ static void free_alt_odbs(struct raw_object_store *o)void raw_object_store_clear (struct raw_object_store *o) {FREE_AND_NULL (o->objectdir); +FREE_AND_NULL (o->alternate_db);free_alt_odbs (o); o->alt_odb_tail =NULL ;repository.c +0
-0
repository.c void repo_set_gitdir(struct repository *repo, · line 65
Quoted 5 lines
repository.c +2
-0@@ -61,6 +61,8 @@ void repo_set_gitdir(struct repository *repo,repo_set_commondir (repo, o->shared_root);expand_base_dir (&repo->objects.objectdir, o->object_dir, repo->commondir,"objects" ); +free (repo->objects.alternate_db); + repo->objects.alternate_db =xstrdup_or_null (o->alternate_db);
Just a note that this only works because the object store is embedded in the repository struct, it isn't a pointer to an object store.
Patch looks good.
Quoted 39 lines
repository.c +0
-0expand_base_dir (&repo->graft_file, o->graft_file, repo->commondir,"info/grafts" ); expand_base_dir(&repo->index_file, o->index_file,repository.h +1
-0@@ -94,6 +94,7 @@ struct set_gitdir_args {const char *object_dir;const char *graft_file;const char *index_file; +const char *alternate_db; }; extern void repo_set_gitdir(struct repository *repo,sha1_file.c +1
-5@@ -673,15 +673,11 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)void prepare_alt_odb (struct repository *r) { -const char *alt; -if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - r->objects.alt_odb_tail = &r->objects.alt_odb_list; -link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (r, r->objects.alternate_db,PATH_SEP ,NULL ,0 );read_info_alternates (r, r->objects.objectdir,0 ); }Signature · 1 line
2.16.1.435.g8f24da2e1a
Signature · 1 line
Brandon Williams
Re: [PATCH v2 3/4] sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
Quoted 14 lines · Brandon Williams
repository.c void repo_set_gitdir(struct repository *repo, · line 65
Quoted 10 lines · Nguyễn Thái Ngọc Duy
repository.c +2
-0@@ -61,6 +61,8 @@ void repo_set_gitdir(struct repository *repo,repo_set_commondir (repo, o->shared_root);expand_base_dir (&repo->objects.objectdir, o->object_dir, repo->commondir,"objects" ); +free (repo->objects.alternate_db); + repo->objects.alternate_db =xstrdup_or_null (o->alternate_db);Just a note that this only works because the object store is embedded in the repository struct, it isn't a pointer to an object store.
It is possible to make it work even if object store is not embedded though. From my point of view, this function is about "give me $GIT_DIR and optionally the where all other parts are, if you ignore default repo layout and tear the repository apart". We could initialize the object store later when it's created and pass the relevant paths to it. Exactly where it's safe to "create and initialize object store" is harder to see now because the whole main repo setup is spread out over many many functions.
Signature · 1 line
Duy
PATCH v2 4/4 repository: delete ignore_env member
In reply to Nguyễn Thái Ngọc DuyThis variable was added because the repo_set_gitdir() was created to cover both submodule and main repos, but these two are initialized a bit differently so ignore_env == 0 means main repo, while ignore_env != 0 is submodules.
Since the difference part (env variables) has been moved out of repo_set_gitdir(), this function works the same way for both repo types and ignore_env is not needed anymore.
--- repository.c | 4 +--- repository.h | 9 --------- 2 files changed, 1 insertion(+), 12 deletions(-)
repository.c +1 -3
@@ -12,7 +12,7 @@ static struct repository the_repo = {NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ], - 0, 0 + 0 };struct repository *the_repository = &the_repo;
@@ -139,8 +139,6 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)struct repository_format format;memset (repo,0 ,sizeof (*repo)); - repo->ignore_env =1 ; -INIT_LIST_HEAD (&repo->objects.packed_git_mru);if (repo_init_gitdir (repo, gitdir))
repository.h +0 -9
@@ -73,15 +73,6 @@ struct repository {const struct git_hash_algo *hash_algo;/* Configurations */ -/* -* Bit used during initialization to indicate if repository state (like -* the location of the 'objectdir') should be read from the -* environment. By default this bit will be set at the begining of -* 'repo_init()' so that all repositories will ignore the environment. -* The exception to this is 'the_repository', which doesn't go through -* the normal 'repo_init()' process. -*/ - unsigned ignore_env:1;/* Indicate if a repository has a different 'commondir' from 'gitdir' */ unsigned different_commondir:1;
Signature · 1 line
2.16.1.435.g8f24da2e1a
Re: [PATCH v2 4/4] repository: delete ignore_env member
Quoted 44 lines · Nguyễn Thái Ngọc Duy
This variable was added because the
repo_set_gitdir()was created to cover both submodule and main repos, but these two are initialized a bit differently soignore_env== 0 means main repo, whileignore_env!= 0 is submodules.Since the difference part (env variables) has been moved out of
repo_set_gitdir(), this function works the same way for both repo types andignore_envis not needed anymore.Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>--- repository.c | 4 +--- repository.h | 9 --------- 2 files changed, 1 insertion(+), 12 deletions(-)repository.c +1
-3@@ -12,7 +12,7 @@ static struct repository the_repo = {NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ], - 0, 0 + 0 };struct repository *the_repository = &the_repo;@@ -139,8 +139,6 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)struct repository_format format;memset (repo,0 ,sizeof (*repo)); - repo->ignore_env =1 ; -INIT_LIST_HEAD (&repo->objects.packed_git_mru);if (repo_init_gitdir (repo, gitdir))repository.h +0
-0@@ -73,15 +73,6 @@ struct repository {const struct git_hash_algo *hash_algo;
repository.h struct repository { · line 84
Quoted 10 lines
repository.h +0
-9/* Configurations */ -/* -* Bit used during initialization to indicate if repository state (like -* the location of the 'objectdir') should be read from the -* environment. By default this bit will be set at the begining of -* 'repo_init()' so that all repositories will ignore the environment. -* The exception to this is 'the_repository', which doesn't go through -* the normal 'repo_init()' process. -*/ - unsigned ignore_env:1;
Nice, glad to see this get a bit less complicated. I'm also very glad I didn't end up using this variable when doing anything else but initializing a repo object otherwise you would have had a bit more difficult time getting rid of it :)
Thanks!
Quoted 6 lines
repository.h +0
-0/* Indicate if a repository has a different 'commondir' from 'gitdir' */ unsigned different_commondir:1;Signature · 1 line
2.16.1.435.g8f24da2e1a
Signature · 1 line
Brandon Williams
Re: [PATCH v2 0/4] Delete ignore_env member in struct repository
In reply to Nguyễn Thái Ngọc Duyenvironment.c static char *expand_namespace(const char *raw_namespace) · line 151
Quoted 8 lines · Nguyễn Thái Ngọc Duy
v2 fixes the incorrect use of consecutive
getenv()and adds a comment to clarify the role ofold_gitdirenvironment.c +3
-0@@ -148,18 +149,34 @@ static char *expand_namespace(const char *raw_namespace) +/* Wrapper of getenv() that returns a strdup value. This value is kept +* in argv to be freed later. +*/
/*
* Comment style.
*/Quoted 7 lines
environment.c +7
-0+static const char *getenv_safe (struct argv_array *argv,const char *name) +{ +const char *value =getenv (name); + +if (!value) +return NULL ; +
environment.c static char *expand_namespace(const char *raw_namespace) · line 171
Quoted 10 lines
environment.c +5
-0+argv_array_push (argv, value); +return argv->argv[argv->argc -1 ]; +} +void setup_git_env (const char *git_dir) {const char *shallow_file;const char *replace_ref_base;struct set_gitdir_args args = {NULL }; +struct argv_array to_free =ARGV_ARRAY_INIT ;
Cute. Another example[1] showing that renaming argv_array to something else might be a good idea.
Quoted 14 lines
environment.c +6
-5- args.shared_root =getenv (GIT_COMMON_DIR_ENVIRONMENT ); - args.object_dir =getenv (DB_ENVIRONMENT ); - args.graft_file =getenv (GRAFT_ENVIRONMENT ); - args.index_file =getenv (INDEX_ENVIRONMENT ); - args.alternate_db =getenv (ALTERNATE_DB_ENVIRONMENT ); + args.shared_root =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); + args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); + args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); + args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT ); + args.alternate_db =getenv_safe (&to_free,ALTERNATE_DB_ENVIRONMENT );repo_set_gitdir (the_repository, git_dir, &args); +argv_array_clear (&to_free);
repository.c void repo_set_gitdir(struct repository *repo, · line 54
Quoted 7 lines
repository.c +5
-0@@ -48,6 +48,11 @@ void repo_set_gitdir(struct repository *repo,const char *gitfile =read_gitfile (root); +/* +* repo->gitdir is saved because the caller could pass "root" +* that also points to repo->gitdir. We want to keep it alive +* until after xstrdup(root). Then we can free it. +*/ char *old_gitdir = repo->gitdir;
Good. This comment makes the reasoning clear.
Quoted 1 line
repository.c +0
-0repo->gitdir =xstrdup (gitfile ? gitfile : root);
Re: [PATCH v2 0/4] Delete ignore_env member in struct repository
In reply to Nguyễn Thái Ngọc DuyQuoted 18 lines · Nguyễn Thái Ngọc Duy
v2 fixes the incorrect use of consecutive
getenv()and adds a comment to clarify the role ofold_gitdirInterdiff:
environment.c +1
-0@@ -14,6 +14,7 @@#include "fmt-merge-msg.h" #include "commit.h" #include "object-store.h" +#include "argv-array.h" int trust_executable_bit =1 ;int trust_ctime =1 ;
environment.c static char *expand_namespace(const char *raw_namespace) · line 154
Quoted 6 lines
environment.c +3
-0@@ -148,18 +149,34 @@ static char *expand_namespace(const char *raw_namespace)return strbuf_detach (&buf,NULL ); } +/* Wrapper of getenv() that returns a strdup value. This value is kept +* in argv to be freed later. +*/
/* comment style, see also Erics reply */
I do not understand why we need to use an argv_array here? I see that the push calls xstrdup and then puts it into the array. So to me this looks like a clever way that we can easily free them all at once using the predefined argv_array_clear() after calling repo_set_gitdir.
That makes sense, though is confusing at first; I would have expected a set_gitdir_args_clear() function that just frees all its strings, whether they are NULL or not.
But given that we are clever with not pushing onto the argv_array in case the getenv returns NULL, this seems to be less work in the usual case of no env variables set.
Ok, I think I like it.
environment.c static char *expand_namespace(const char *raw_namespace) · line 155
Quoted 1 line
environment.c +1
-0+static const char *getenv_safe (struct argv_array *argv,const char *name)
I would have expected that we already have such a (or similar) helper in wrapper.c, but it turns out we always take care of getenv manually, and we do not have a wrapper yet.
So only the comment style remains as a nit.
Thanks,
Stefan
v3 cover Delete ignore_env member in struct repository
In reply to Nguyễn Thái Ngọc Duyv3 fixes comment style. Also since Brandon raised a question about shared_root, it's obviously not a good name, so I renamed it to commondir.
I still keep the delete patch 2/4, but I move the repo_setup_env() deletion back to 1/4 so all env logic is in one patch (the introduction of new helper functions in 1/4 and deletion in 2/4 are still diff noise if 2/4 is completely merged back).
Interdiff:
environment.c +3 -2
@@ -149,7 +149,8 @@ static char *expand_namespace(const char *raw_namespace)return strbuf_detach (&buf,NULL ); } -/* Wrapper of getenv() that returns a strdup value. This value is kept +/* +* Wrapper of getenv() that returns a strdup value. This value is kept * in argv to be freed later. */ static const char *getenv_safe (struct argv_array *argv,const char *name)
@@ -170,7 +171,7 @@ void setup_git_env(const char *git_dir)struct set_gitdir_args args = {NULL };struct argv_array to_free =ARGV_ARRAY_INIT ; - args.shared_root =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); + args.commondir =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT );
repository.c +4 -4
@@ -27,15 +27,15 @@ static void expand_base_dir(char **out, const char *in, }static void repo_set_commondir (struct repository *repo, -const char *shared_root) +const char *commondir) {struct strbuf sb =STRBUF_INIT ;free (repo->commondir); -if (shared_root) { +if (commondir) { repo->different_commondir =1 ; - repo->commondir =xstrdup (shared_root); + repo->commondir =xstrdup (commondir);return ; }
@@ -58,7 +58,7 @@ void repo_set_gitdir(struct repository *repo, repo->gitdir =xstrdup (gitfile ? gitfile : root);free (old_gitdir); -repo_set_commondir (repo, o->shared_root); +repo_set_commondir (repo, o->commondir);expand_base_dir (&repo->objects.objectdir, o->object_dir, repo->commondir,"objects" );free (repo->objects.alternate_db);
repository.h +1 -1
@@ -81,7 +81,7 @@ struct repository {extern struct repository *the_repository;struct set_gitdir_args { -const char *shared_root; +const char *commondir;const char *object_dir;const char *graft_file;const char *index_file;
Nguyễn Thái Ngọc Duy (4):
repository.c: move env-related setup code back to environment.c repository.c: delete dead functions
sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
repository: delete ignore_env member
cache.h | 2 +- environment.c | 31 +++++++++++++++++-- object-store.h | 5 ++- object.c | 1 + repository.c | 84 ++++++++++++++++++++++++-------------------------- repository.h | 21 +++++++------ setup.c | 3 +- sha1_file.c | 6 +--- 8 files changed, 87 insertions(+), 66 deletions(-)
Signature · 1 line
2.16.1.399.g632f88eed1
PATCH v3 1/4 repository.c: move env-related setup code back to environment.c
It does not make sense that generic repository code contains handling of environment variables, which are specific for the main repository only. Refactor repo_set_gitdir() function to take $GIT_DIR and optionally _all_ other customizable paths. These optional paths can be NULL and will be calculated according to the default directory layout.
Note that some dead functions are left behind to reduce diff noise. They will be deleted in the next patch.
--- cache.h | 2 +- environment.c | 30 +++++++++++++++++++++++--- repository.c | 59 ++++++++++++++++++++++++++++++++++----------------- repository.h | 11 +++++++++- setup.c | 3 +-- 5 files changed, 79 insertions(+), 26 deletions(-)
cache.h +1 -1
@@ -459,7 +459,7 @@ static inline enum object_type object_type(unsigned int mode)*/ extern const char *const local_repo_env[]; -extern void setup_git_env (void ); +extern void setup_git_env (const char *git_dir);/* * Returns true iff we have a configured git repository (either via
environment.c +27 -3
@@ -14,6 +14,7 @@#include "fmt-merge-msg.h" #include "commit.h" #include "object-store.h" +#include "argv-array.h" int trust_executable_bit =1 ;int trust_ctime =1 ;
@@ -148,10 +149,34 @@ static char *expand_namespace(const char *raw_namespace)return strbuf_detach (&buf,NULL ); } -void setup_git_env (void ) +/* +* Wrapper of getenv() that returns a strdup value. This value is kept +* in argv to be freed later. +*/ +static const char *getenv_safe (struct argv_array *argv,const char *name) +{ +const char *value =getenv (name); + +if (!value) +return NULL ; + +argv_array_push (argv, value); +return argv->argv[argv->argc -1 ]; +} + +void setup_git_env (const char *git_dir) {const char *shallow_file;const char *replace_ref_base; +struct set_gitdir_args args = {NULL }; +struct argv_array to_free =ARGV_ARRAY_INIT ; + + args.commondir =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); + args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); + args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); + args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT ); +repo_set_gitdir (the_repository, git_dir, &args); +argv_array_clear (&to_free);if (getenv (NO_REPLACE_OBJECTS_ENVIRONMENT )) check_replace_refs =0 ;
@@ -301,8 +326,7 @@ int set_git_dir(const char *path) {if (setenv (GIT_DIR_ENVIRONMENT , path,1 ))return error ("Could not set GIT_DIR to '%s'" , path); -repo_set_gitdir (the_repository, path); -setup_git_env (); +setup_git_env (path);return 0 ; }
repository.c +40 -19
88 lines not on this page · raw message
repository.h +10 -1
@@ -89,7 +89,16 @@ struct repository {extern struct repository *the_repository; -extern void repo_set_gitdir (struct repository *repo,const char *path); +struct set_gitdir_args { +const char *commondir; +const char *object_dir; +const char *graft_file; +const char *index_file; +}; + +extern void repo_set_gitdir (struct repository *repo, +const char *root, +const struct set_gitdir_args *optional);extern void repo_set_worktree (struct repository *repo,const char *path);extern void repo_set_hash_algo (struct repository *repo,int algo);extern int repo_init (struct repository *repo,const char *gitdir,const char *worktree);
setup.c +1 -2
@@ -1116,8 +1116,7 @@ const char *setup_git_directory_gently(int *nongit_ok)const char *gitdir =getenv (GIT_DIR_ENVIRONMENT );if (!gitdir) gitdir =DEFAULT_GIT_DIR_ENVIRONMENT ; -repo_set_gitdir (the_repository, gitdir); -setup_git_env (); +setup_git_env (gitdir); }if (startup_info->have_repository)repo_set_hash_algo (the_repository, repo_fmt.hash_algo);
Signature · 1 line
2.16.1.399.g632f88eed1
PATCH v3 2/4 repository.c: delete dead functions
In reply to Nguyễn Thái Ngọc Duy--- repository.c | 25 ------------------------- 1 file changed, 25 deletions(-)
repository.c +0 -25
@@ -16,31 +16,6 @@ static struct repository the_repo = { };struct repository *the_repository = &the_repo; -static char *git_path_from_env (const char *envvar,const char *git_dir, -const char *path,int fromenv) -{ -if (fromenv) { -const char *value =getenv (envvar); -if (value) -return xstrdup (value); - } - -return xstrfmt ("%s/%s" , git_dir, path); -} - -static int find_common_dir (struct strbuf *sb,const char *gitdir,int fromenv) -{ -if (fromenv) { -const char *value =getenv (GIT_COMMON_DIR_ENVIRONMENT ); -if (value) { -strbuf_addstr (sb, value); -return 1 ; - } - } - -return get_common_dir_noenv (sb, gitdir); -} -static void expand_base_dir (char **out,const char *in,const char *base_dir,const char *def_in) {
Signature · 1 line
2.16.1.399.g632f88eed1
PATCH v3 3/4 sha1_file.c: move delayed getenv(altdb) back to setup_git_env()
In reply to Nguyễn Thái Ngọc Duygetenv() is supposed to work on the main repository only. This delayed getenv() code in sha1_file.c makes it more difficult to convert sha1_file.c to a generic object store that could be used by both submodule and main repositories.
Move the getenv() back in setup_git_env() where other env vars are also fetched.
--- environment.c | 1 + object-store.h | 5 ++++- object.c | 1 + repository.c | 2 ++ repository.h | 1 + sha1_file.c | 6 +----- 6 files changed, 10 insertions(+), 6 deletions(-)
environment.c +1 -0
@@ -175,6 +175,7 @@ void setup_git_env(const char *git_dir) args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT ); + args.alternate_db =getenv_safe (&to_free,ALTERNATE_DB_ENVIRONMENT );repo_set_gitdir (the_repository, git_dir, &args);argv_array_clear (&to_free);
object-store.h +4 -1
@@ -87,6 +87,9 @@ struct raw_object_store {*/ char *objectdir; +/* Path to extra alternate object database if not NULL */ +char *alternate_db; +struct packed_git *packed_git;/* A most-recently-used ordered version of the packed_git list. */ struct list_head packed_git_mru;
@@ -109,7 +112,7 @@ struct raw_object_store { unsigned packed_git_initialized : 1; }; -#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 } +#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);
object.c +1 -0
@@ -464,6 +464,7 @@ static void free_alt_odbs(struct raw_object_store *o)void raw_object_store_clear (struct raw_object_store *o) {FREE_AND_NULL (o->objectdir); +FREE_AND_NULL (o->alternate_db);free_alt_odbs (o); o->alt_odb_tail =NULL ;
repository.c +2 -0
@@ -61,6 +61,8 @@ void repo_set_gitdir(struct repository *repo,repo_set_commondir (repo, o->commondir);expand_base_dir (&repo->objects.objectdir, o->object_dir, repo->commondir,"objects" ); +free (repo->objects.alternate_db); + repo->objects.alternate_db =xstrdup_or_null (o->alternate_db);expand_base_dir (&repo->graft_file, o->graft_file, repo->commondir,"info/grafts" ); expand_base_dir(&repo->index_file, o->index_file,
repository.h +1 -0
@@ -94,6 +94,7 @@ struct set_gitdir_args {const char *object_dir;const char *graft_file;const char *index_file; +const char *alternate_db; }; extern void repo_set_gitdir(struct repository *repo,
sha1_file.c +1 -5
@@ -673,15 +673,11 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)void prepare_alt_odb (struct repository *r) { -const char *alt; -if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - r->objects.alt_odb_tail = &r->objects.alt_odb_list; -link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (r, r->objects.alternate_db,PATH_SEP ,NULL ,0 );read_info_alternates (r, r->objects.objectdir,0 ); }
Signature · 1 line
2.16.1.399.g632f88eed1
PATCH v3 4/4 repository: delete ignore_env member
In reply to Nguyễn Thái Ngọc DuyThis variable was added because the repo_set_gitdir() was created to cover both submodule and main repos, but these two are initialized a bit differently so ignore_env == 0 means main repo, while ignore_env != 0 is submodules.
Since the difference part (env variables) has been moved out of repo_set_gitdir(), this function works the same way for both repo types and ignore_env is not needed anymore.
--- repository.c | 4 +--- repository.h | 9 --------- 2 files changed, 1 insertion(+), 12 deletions(-)
repository.c +1 -3
@@ -12,7 +12,7 @@ static struct repository the_repo = {NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ], - 0, 0 + 0 };struct repository *the_repository = &the_repo;
@@ -139,8 +139,6 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)struct repository_format format;memset (repo,0 ,sizeof (*repo)); - repo->ignore_env =1 ; -INIT_LIST_HEAD (&repo->objects.packed_git_mru);if (repo_init_gitdir (repo, gitdir))
repository.h +0 -9
@@ -73,15 +73,6 @@ struct repository {const struct git_hash_algo *hash_algo;/* Configurations */ -/* -* Bit used during initialization to indicate if repository state (like -* the location of the 'objectdir') should be read from the -* environment. By default this bit will be set at the begining of -* 'repo_init()' so that all repositories will ignore the environment. -* The exception to this is 'the_repository', which doesn't go through -* the normal 'repo_init()' process. -*/ - unsigned ignore_env:1;/* Indicate if a repository has a different 'commondir' from 'gitdir' */ unsigned different_commondir:1;
Signature · 1 line
2.16.1.399.g632f88eed1
Re: [PATCH v3 0/4] Delete ignore_env member in struct repository
In reply to Nguyễn Thái Ngọc DuyQuoted 3 lines · Nguyễn Thái Ngọc Duy
v3 fixes comment style. Also since Brandon raised a question about
shared_root, it's obviously not a good name, so I renamed it to commondir.
Thanks, its a bit clearer and more consistent with the rest of the terminology we already have :)
Quoted 96 lines
I still keep the delete patch 2/4, but I move the
repo_setup_env()deletion back to 1/4 so all env logic is in one patch (the introduction of new helper functions in 1/4 and deletion in 2/4 are still diff noise if 2/4 is completely merged back).Interdiff:
environment.c +3
-2@@ -149,7 +149,8 @@ static char *expand_namespace(const char *raw_namespace)return strbuf_detach (&buf,NULL ); } -/* Wrapper of getenv() that returns a strdup value. This value is kept +/* +* Wrapper of getenv() that returns a strdup value. This value is kept * in argv to be freed later. */ static const char *getenv_safe (struct argv_array *argv,const char *name)@@ -170,7 +171,7 @@ void setup_git_env(const char *git_dir)struct set_gitdir_args args = {NULL };struct argv_array to_free =ARGV_ARRAY_INIT ; - args.shared_root =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); + args.commondir =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT );repository.c +4
-4@@ -27,15 +27,15 @@ static void expand_base_dir(char **out, const char *in, }static void repo_set_commondir (struct repository *repo, -const char *shared_root) +const char *commondir) {struct strbuf sb =STRBUF_INIT ;free (repo->commondir); -if (shared_root) { +if (commondir) { repo->different_commondir =1 ; - repo->commondir =xstrdup (shared_root); + repo->commondir =xstrdup (commondir);return ; }@@ -58,7 +58,7 @@ void repo_set_gitdir(struct repository *repo, repo->gitdir =xstrdup (gitfile ? gitfile : root);free (old_gitdir); -repo_set_commondir (repo, o->shared_root); +repo_set_commondir (repo, o->commondir);expand_base_dir (&repo->objects.objectdir, o->object_dir, repo->commondir,"objects" );free (repo->objects.alternate_db);repository.h +1
-1@@ -81,7 +81,7 @@ struct repository {extern struct repository *the_repository;struct set_gitdir_args { -const char *shared_root; +const char *commondir;const char *object_dir;const char *graft_file;const char *index_file;Nguyễn Thái Ngọc Duy (4):
repository.c: move env-related setup code back toenvironment.crepository.c: delete dead functions
sha1_file.c: move delayed getenv(altdb) back tosetup_git_env()
repository: deleteignore_envmembercache.h | 2 +- environment.c | 31 +++++++++++++++++-- object-store.h | 5 ++- object.c | 1 + repository.c | 84 ++++++++++++++++++++++++-------------------------- repository.h | 21 +++++++------ setup.c | 3 +- sha1_file.c | 6 +--- 8 files changed, 87 insertions(+), 66 deletions(-)Signature · 1 line
2.16.1.399.g632f88eed1
Signature · 1 line
Brandon Williams
Re: [PATCH v3 0/4] Delete ignore_env member in struct repository
In reply to Nguyễn Thái Ngọc DuyQuoted 8 lines · Nguyễn Thái Ngọc Duy
v3 fixes comment style. Also since Brandon raised a question about
shared_root, it's obviously not a good name, so I renamed it to commondir.I still keep the delete patch 2/4, but I move the
repo_setup_env()deletion back to 1/4 so all env logic is in one patch (the introduction of new helper functions in 1/4 and deletion in 2/4 are still diff noise if 2/4 is completely merged back).
Thanks. After replacing sb/object-store with the last one posted and re-tweaking resolution of conflicts with ds/commit-graph that is in flight, I think I can squeeze this topic in. Among the three, I find that this kind of clean-up is the most interesting/urgent thing we need before we can move forward ;-)
Quoted 87 lines
Interdiff:
environment.c +3
-2@@ -149,7 +149,8 @@ static char *expand_namespace(const char *raw_namespace)return strbuf_detach (&buf,NULL ); } -/* Wrapper of getenv() that returns a strdup value. This value is kept +/* +* Wrapper of getenv() that returns a strdup value. This value is kept * in argv to be freed later. */ static const char *getenv_safe (struct argv_array *argv,const char *name)@@ -170,7 +171,7 @@ void setup_git_env(const char *git_dir)struct set_gitdir_args args = {NULL };struct argv_array to_free =ARGV_ARRAY_INIT ; - args.shared_root =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); + args.commondir =getenv_safe (&to_free,GIT_COMMON_DIR_ENVIRONMENT ); args.object_dir =getenv_safe (&to_free,DB_ENVIRONMENT ); args.graft_file =getenv_safe (&to_free,GRAFT_ENVIRONMENT ); args.index_file =getenv_safe (&to_free,INDEX_ENVIRONMENT );repository.c +4
-4@@ -27,15 +27,15 @@ static void expand_base_dir(char **out, const char *in, }static void repo_set_commondir (struct repository *repo, -const char *shared_root) +const char *commondir) {struct strbuf sb =STRBUF_INIT ;free (repo->commondir); -if (shared_root) { +if (commondir) { repo->different_commondir =1 ; - repo->commondir =xstrdup (shared_root); + repo->commondir =xstrdup (commondir);return ; }@@ -58,7 +58,7 @@ void repo_set_gitdir(struct repository *repo, repo->gitdir =xstrdup (gitfile ? gitfile : root);free (old_gitdir); -repo_set_commondir (repo, o->shared_root); +repo_set_commondir (repo, o->commondir);expand_base_dir (&repo->objects.objectdir, o->object_dir, repo->commondir,"objects" );free (repo->objects.alternate_db);repository.h +1
-1@@ -81,7 +81,7 @@ struct repository {extern struct repository *the_repository;struct set_gitdir_args { -const char *shared_root; +const char *commondir;const char *object_dir;const char *graft_file;const char *index_file;Nguyễn Thái Ngọc Duy (4):
repository.c: move env-related setup code back toenvironment.crepository.c: delete dead functions
sha1_file.c: move delayed getenv(altdb) back tosetup_git_env()
repository: deleteignore_envmembercache.h | 2 +- environment.c | 31 +++++++++++++++++-- object-store.h | 5 ++- object.c | 1 + repository.c | 84 ++++++++++++++++++++++++-------------------------- repository.h | 21 +++++++------ setup.c | 3 +- sha1_file.c | 6 +--- 8 files changed, 87 insertions(+), 66 deletions(-)
PATCH 15/26 sha1_file: add repository argument to sha1_file_name
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
Add a repository argument to allow sha1_file_name callers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
While at it, move the declaration to object-store.h, where it should be easier to find.
--- cache.h | 8 -------- http-walker.c | 3 ++- http.c | 5 +++-- object-store.h | 9 +++++++++ sha1_file.c | 11 ++++++----- 5 files changed, 20 insertions(+), 16 deletions(-)
cache.h +0 -8
@@ -956,14 +956,6 @@ extern void check_repository_format(void);#define DATA_CHANGED 0x0020#define TYPE_CHANGED 0x0040 -/* -* Return the name of the file in the local object database that would -* be used to store a loose object with the specified sha1. The -* return value is a pointer to a statically allocated buffer that is -* overwritten each time the function is called. -*/ -extern const char *sha1_file_name (const unsigned char *sha1); -/* * Return an abbreviated sha1 unique within this repository's object database. * The result will be at least `len` characters long, and will be NUL
http-walker.c +2 -1
@@ -1,4 +1,5 @@#include "cache.h" +#include "repository.h" #include "commit.h" #include "walker.h" #include "http.h"
@@ -546,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1) ret =error ("File %s has bad hash" , hex); }else if (req->rename <0 ) { ret =error ("unable to write sha1 filename %s" , -sha1_file_name (req->sha1)); +sha1_file_name (the_repository, req->sha1)); }release_http_object_request (req);
http.c +3 -2
@@ -2181,7 +2181,7 @@ struct http_object_request *new_http_object_request(const char *base_url,hashcpy (freq->sha1, sha1); freq->localfile =-1 ; - filename =sha1_file_name (sha1); + filename =sha1_file_name (the_repository, sha1);snprintf (freq->tmpfile,sizeof (freq->tmpfile),"%s.temp" , filename);
@@ -2329,7 +2329,8 @@ int finish_http_object_request(struct http_object_request *freq)return -1 ; } freq->rename = -finalize_object_file (freq->tmpfile,sha1_file_name (freq->sha1)); +finalize_object_file (freq->tmpfile, +sha1_file_name (the_repository, freq->sha1));return freq->rename; }
object-store.h +9 -0
@@ -61,6 +61,15 @@ struct packed_git { char pack_name[FLEX_ARRAY]; /* more */ }; +/* +* Return the name of the file in a repository's local object database +* that would be used to store a loose object with the specified sha1. +* The return value is a pointer to a statically allocated buffer that +* is overwritten each time the function is called. +*/ +#define sha1_file_name (r, s) sha1_file_name_##r(s) +const char *sha1_file_name_the_repository (const unsigned char *sha1); +void prepare_alt_odb (struct repository *r);#endif /* OBJECT_STORE_H */
sha1_file.c +6 -5
@@ -323,7 +323,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1) } } -const char *sha1_file_name (const unsigned char *sha1) +const char *sha1_file_name_the_repository (const unsigned char *sha1) {static struct strbuf buf =STRBUF_INIT ;
@@ -721,7 +721,8 @@ int check_and_freshen_file(const char *fn, int freshen)static int check_and_freshen_local (const unsigned char *sha1,int freshen) { -return check_and_freshen_file (sha1_file_name (sha1), freshen); +return check_and_freshen_file (sha1_file_name (the_repository, sha1), + freshen); }static int check_and_freshen_nonlocal (const unsigned char *sha1,int freshen)
@@ -878,7 +879,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st, {struct alternate_object_database *alt; - *path =sha1_file_name (sha1); + *path =sha1_file_name (the_repository, sha1);if (!lstat (*path, st))return 0 ;
@@ -903,7 +904,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)struct alternate_object_database *alt;int most_interesting_errno; - *path =sha1_file_name (sha1); + *path =sha1_file_name (the_repository, sha1); fd =git_open (*path);if (fd >=0 )return fd;
@@ -1568,7 +1569,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,git_SHA_CTX c;unsigned char parano_sha1[20 ];static struct strbuf tmp_file =STRBUF_INIT ; -const char *filename =sha1_file_name (sha1); +const char *filename =sha1_file_name (the_repository, sha1); fd =create_tmpfile (&tmp_file, filename);if (fd <0 ) {
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 16/26 sha1_file: add repository argument to stat_sha1_file
In reply to Stefan BellerAdd a repository argument to allow the stat_sha1_file caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
sha1_file.c +5 -4
@@ -874,8 +874,9 @@ int git_open_cloexec(const char *name, int flags)* Note that it may point to static storage and is only valid until another * call to sha1_file_name(), etc. */ -static int stat_sha1_file (const unsigned char *sha1,struct stat *st, -const char **path) +#define stat_sha1_file (r, s, st, p) stat_sha1_file_##r(s, st, p) +static int stat_sha1_file_the_repository (const unsigned char *sha1, +struct stat *st,const char **path) {struct alternate_object_database *alt;
@@ -1173,7 +1174,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {const char *path;struct stat st; -if (stat_sha1_file (sha1, &st, &path) <0 ) +if (stat_sha1_file (the_repository, sha1, &st, &path) <0 )return -1 ;if (oi->disk_sizep) *oi->disk_sizep = st.st_size;
@@ -1372,7 +1373,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,die ("replacement %s not found for %s" ,sha1_to_hex (repl),sha1_to_hex (sha1)); -if (!stat_sha1_file (repl, &st, &path)) +if (!stat_sha1_file (the_repository, repl, &st, &path))die ("loose object %s (stored in %s) is corrupt" ,sha1_to_hex (repl), path);
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 17/26 sha1_file: add repository argument to open_sha1_file
In reply to Stefan BellerAdd a repository argument to allow the open_sha1_file caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
sha1_file.c +4 -2
@@ -899,7 +899,9 @@ static int stat_sha1_file_the_repository(const unsigned char *sha1,* Like stat_sha1_file(), but actually open the object and return the * descriptor. See the caveats on the "path" parameter above. */ -static int open_sha1_file (const unsigned char *sha1,const char **path) +#define open_sha1_file (r, s, p) open_sha1_file_##r(s, p) +static int open_sha1_file_the_repository (const unsigned char *sha1, +const char **path) {int fd;struct alternate_object_database *alt;
@@ -938,7 +940,7 @@ static void *map_sha1_file_1(const char *path,if (path) fd =git_open (path);else - fd =open_sha1_file (sha1, &path); + fd =open_sha1_file (the_repository, sha1, &path); map =NULL ;if (fd >=0 ) {struct stat st;
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 18/26 sha1_file: add repository argument to map_sha1_file_1
In reply to Stefan BellerAdd a repository argument to allow the map_sha1_file_1 caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
sha1_file.c +6 -5
@@ -930,9 +930,10 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,* Map the loose object at "path" if it is not NULL, or the path found by * searching for a loose object named "sha1". */ -static void *map_sha1_file_1 (const char *path, -const unsigned char *sha1, -unsigned long *size) +#define map_sha1_file_1 (r, p, s, si) map_sha1_file_1_##r(p, s, si) +static void *map_sha1_file_1_the_repository (const char *path, +const unsigned char *sha1, +unsigned long *size) {void *map;int fd;
@@ -961,7 +962,7 @@ static void *map_sha1_file_1(const char *path,void *map_sha1_file (const unsigned char *sha1,unsigned long *size) { -return map_sha1_file_1 (NULL , sha1, size); +return map_sha1_file_1 (the_repository,NULL , sha1, size); } static int unpack_sha1_short_header(git_zstream *stream,
@@ -2173,7 +2174,7 @@ int read_loose_object(const char *path, *contents =NULL ; - map =map_sha1_file_1 (path,NULL , &mapsize); + map =map_sha1_file_1 (the_repository, path,NULL , &mapsize);if (!map) {error_errno ("unable to mmap %s" , path); goto out;
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 19/26 sha1_file: add repository argument to map_sha1_file
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
Add a repository argument to allow map_sha1_file callers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
While at it, move the declaration to object-store.h, where it should be easier to find.
--- cache.h | 1 - object-store.h | 2 ++ sha1_file.c | 4 ++-- streaming.c | 5 ++++- 4 files changed, 8 insertions(+), 4 deletions(-)
cache.h +0 -1
@@ -1237,7 +1237,6 @@ extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned cextern int force_object_loose (const unsigned char *sha1,time_t mtime);extern int git_open_cloexec (const char *name,int flags);#define git_open (name) git_open_cloexec(name, O_RDONLY) -extern void *map_sha1_file (const unsigned char *sha1,unsigned long *size);extern int unpack_sha1_header (git_zstream *stream,unsigned char *map,unsigned long mapsize,void *buffer,unsigned long bufsiz);extern int parse_sha1_header (const char *hdr,unsigned long *sizep);
object-store.h +2 -0
@@ -69,6 +69,8 @@ struct packed_git {*/ #define sha1_file_name (r, s) sha1_file_name_##r(s)const char *sha1_file_name_the_repository (const unsigned char *sha1); +#define map_sha1_file (r, s, sz) map_sha1_file_##r(s, sz) +void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size);void prepare_alt_odb (struct repository *r);
sha1_file.c +2 -2
@@ -960,7 +960,7 @@ static void *map_sha1_file_1_the_repository(const char *path,return map; } -void *map_sha1_file (const unsigned char *sha1,unsigned long *size) +void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size) {return map_sha1_file_1 (the_repository,NULL , sha1, size); }
@@ -1184,7 +1184,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,return 0 ; } - map =map_sha1_file (sha1, &mapsize); + map =map_sha1_file (the_repository, sha1, &mapsize);if (!map)return -1 ;
streaming.c +4 -1
@@ -3,6 +3,8 @@*/ #include "cache.h" #include "streaming.h" +#include "repository.h" +#include "object-store.h" #include "packfile.h" enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {static open_method_decl (loose) { - st->u.loose.mapped =map_sha1_file (sha1, &st->u.loose.mapsize); + st->u.loose.mapped =map_sha1_file (the_repository, + sha1, &st->u.loose.mapsize);if (!st->u.loose.mapped)return -1 ; if ((unpack_sha1_header(&st->z,
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 20/26 sha1_file: add repository argument to sha1_loose_object_info
In reply to Stefan BellerAdd a repository argument to allow the sha1_loose_object_info caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
sha1_file.c +5 -4
@@ -1151,9 +1151,10 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -static int sha1_loose_object_info (const unsigned char *sha1, -struct object_info *oi, -int flags) +#define sha1_loose_object_info (r, s, o, f) sha1_loose_object_info_##r(s, o, f) +static int sha1_loose_object_info_the_repository (const unsigned char *sha1, +struct object_info *oi, +int flags) {int status =0 ;unsigned long mapsize;
@@ -1266,7 +1267,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,if (!find_pack_entry (real, &e)) {/* Most likely it's a loose object. */ -if (!sha1_loose_object_info (real, oi, flags)) +if (!sha1_loose_object_info (the_repository, real, oi, flags))return 0 ;/* Not a loose object; someone else may have just packed it. */
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 21/26 sha1_file: allow sha1_file_name to handle arbitrary repositories
In reply to Stefan Beller--- object-store.h | 3 +-- sha1_file.c | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-)
object-store.h +1 -2
@@ -67,8 +67,7 @@ struct packed_git {* The return value is a pointer to a statically allocated buffer that * is overwritten each time the function is called. */ -#define sha1_file_name (r, s) sha1_file_name_##r(s) -const char *sha1_file_name_the_repository (const unsigned char *sha1); +const char *sha1_file_name (struct repository *r,const unsigned char *sha1);#define map_sha1_file (r, s, sz) map_sha1_file_##r(s, sz)void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size);
sha1_file.c +2 -2
@@ -323,12 +323,12 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1) } } -const char *sha1_file_name_the_repository (const unsigned char *sha1) +const char *sha1_file_name (struct repository *r,const unsigned char *sha1) {static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -strbuf_addf (&buf,"%s/" ,get_object_directory ()); +strbuf_addf (&buf,"%s/" , r->objects.objectdir);fill_sha1_path (&buf, sha1);return buf.buf;
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 22/26 sha1_file: allow stat_sha1_file to handle arbitrary repositories
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -874,19 +874,18 @@ int git_open_cloexec(const char *name, int flags)* Note that it may point to static storage and is only valid until another * call to sha1_file_name(), etc. */ -#define stat_sha1_file (r, s, st, p) stat_sha1_file_##r(s, st, p) -static int stat_sha1_file_the_repository (const unsigned char *sha1, -struct stat *st,const char **path) +static int stat_sha1_file (struct repository *r,const unsigned char *sha1, +struct stat *st,const char **path) {struct alternate_object_database *alt; - *path =sha1_file_name (the_repository, sha1); + *path =sha1_file_name (r, sha1);if (!lstat (*path, st))return 0 ; -prepare_alt_odb (the_repository); +prepare_alt_odb (r); errno =ENOENT ; -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +for (alt = r->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);if (!lstat (*path, st))return 0 ;
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 23/26 sha1_file: allow open_sha1_file to handle arbitrary repositories
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -898,22 +898,21 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,* Like stat_sha1_file(), but actually open the object and return the * descriptor. See the caveats on the "path" parameter above. */ -#define open_sha1_file (r, s, p) open_sha1_file_##r(s, p) -static int open_sha1_file_the_repository (const unsigned char *sha1, -const char **path) +static int open_sha1_file (struct repository *r, +const unsigned char *sha1,const char **path) {int fd;struct alternate_object_database *alt;int most_interesting_errno; - *path =sha1_file_name (the_repository, sha1); + *path =sha1_file_name (r, sha1); fd =git_open (*path);if (fd >=0 )return fd; most_interesting_errno = errno; -prepare_alt_odb (the_repository); -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +prepare_alt_odb (r); +for (alt = r->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);if (fd >=0 )
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 24/26 sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
--- sha1_file.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
sha1_file.c +3 -5
@@ -928,10 +928,8 @@ static int open_sha1_file(struct repository *r,* Map the loose object at "path" if it is not NULL, or the path found by * searching for a loose object named "sha1". */ -#define map_sha1_file_1 (r, p, s, si) map_sha1_file_1_##r(p, s, si) -static void *map_sha1_file_1_the_repository (const char *path, -const unsigned char *sha1, -unsigned long *size) +static void *map_sha1_file_1 (struct repository *r,const char *path, +const unsigned char *sha1,unsigned long *size) {void *map;int fd;
@@ -939,7 +937,7 @@ static void *map_sha1_file_1_the_repository(const char *path,if (path) fd =git_open (path);else - fd =open_sha1_file (the_repository, sha1, &path); + fd =open_sha1_file (r, sha1, &path); map =NULL ;if (fd >=0 ) {struct stat st;
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 25/26 sha1_file: allow map_sha1_file to handle arbitrary repositories
In reply to Stefan Beller--- object-store.h | 3 +-- sha1_file.c | 5 +++-- 2 files changed, 4 insertions(+), 4 deletions(-)
object-store.h +1 -2
@@ -68,8 +68,7 @@ struct packed_git {* is overwritten each time the function is called. */ const char *sha1_file_name (struct repository *r,const unsigned char *sha1); -#define map_sha1_file (r, s, sz) map_sha1_file_##r(s, sz) -void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size); +extern void *map_sha1_file (struct repository *r,const unsigned char *sha1,unsigned long *size);void prepare_alt_odb (struct repository *r);
sha1_file.c +3 -2
@@ -956,9 +956,10 @@ static void *map_sha1_file_1(struct repository *r, const char *path,return map; } -void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size) +void *map_sha1_file (struct repository *r, +const unsigned char *sha1,unsigned long *size) { -return map_sha1_file_1 (the_repository,NULL , sha1, size); +return map_sha1_file_1 (r,NULL , sha1, size); } static int unpack_sha1_short_header(git_zstream *stream,
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
PATCH 26/26 sha1_file: allow sha1_loose_object_info to handle arbitrary repositories
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -1148,10 +1148,9 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -#define sha1_loose_object_info (r, s, o, f) sha1_loose_object_info_##r(s, o, f) -static int sha1_loose_object_info_the_repository (const unsigned char *sha1, -struct object_info *oi, -int flags) +static int sha1_loose_object_info (struct repository *r, +const unsigned char *sha1, +struct object_info *oi,int flags) {int status =0 ;unsigned long mapsize;
@@ -1175,14 +1174,14 @@ static int sha1_loose_object_info_the_repository(const unsigned char *sha1,if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {const char *path;struct stat st; -if (stat_sha1_file (the_repository, sha1, &st, &path) <0 ) +if (stat_sha1_file (r, sha1, &st, &path) <0 )return -1 ;if (oi->disk_sizep) *oi->disk_sizep = st.st_size;return 0 ; } - map =map_sha1_file (the_repository, sha1, &mapsize); + map =map_sha1_file (r, sha1, &mapsize);if (!map)return -1 ;
Signature · 1 line
2.16.1.73.ga2c3e9663f.dirty
Quoted 1 line · Stefan Beller
I developed this series [...] on top of current master.
also available at https://github.com/stefanbeller/git/tree/object-store-part1
Quoted 4 lines · Stefan Beller
This is a real take on the first part of the recent RFC[1].
...
Quoted 8 lines
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the
ignore_envflag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.
Second proposal :) How about you store ignore_env in raw_object_store? This would not be the first time an object has some configuration passed in at construction time. And it has a "constructor" now, raw_object_store_init() (I probably should merge _setup in it too)
The core changes look like this. I have a full commit on top of your series [1] that keeps sha1_file.c functions take 'struct raw_object_store' instead.
[1] https://github.com/pclouds/git/tree/object-store-part1
-- 8< --
object-store.h +4 -0
@@ -34,9 +34,13 @@ struct raw_object_store {* packs. */ unsigned packed_git_initialized : 1; + + unsigned ignore_env : 1; };#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0, 0, 0 } +void raw_object_store_init (struct raw_object_store *ros,struct repository *r); +void raw_object_store_setup (struct raw_object_store *ros,char *);void raw_object_store_clear (struct raw_object_store *o);struct packed_git {
object.c +14 -0
@@ -461,6 +461,20 @@ static void free_alt_odbs(struct raw_object_store *o) } } +void raw_object_store_init (struct raw_object_store *o, +struct repository *r) +{ +/* FIXME: memset? */ + o->ignore_env = r->ignore_env; +} + +void raw_object_store_setup (struct raw_object_store *o, +char *objectdir) +{ +free (o->objectdir); + o->objectdir = objectdir; +} +void raw_object_store_clear (struct raw_object_store *o) {free (o->objectdir);
repository.c +5 -3
@@ -49,9 +49,9 @@ static void repo_setup_env(struct repository *repo) !repo->ignore_env);free (repo->commondir); repo->commondir =strbuf_detach (&sb,NULL ); -free (repo->objects.objectdir); - repo->objects.objectdir =git_path_from_env (DB_ENVIRONMENT , repo->commondir, -"objects" , !repo->ignore_env); +raw_object_store_setup (&repo->objects, +git_path_from_env (DB_ENVIRONMENT , repo->commondir, +"objects" , !repo->ignore_env));free (repo->graft_file); repo->graft_file =git_path_from_env (GRAFT_ENVIRONMENT , repo->commondir,"info/grafts" , !repo->ignore_env);
@@ -142,6 +142,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree) repo->ignore_env =1 ; +raw_object_store_init (&repo->objects, repo); +if (repo_init_gitdir (repo, gitdir)) goto error;
sha1_file.c +2 -2
@@ -677,14 +677,14 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb (struct repository *r) +void prepare_alt_odb (struct raw_object_store *ros) {if (r->objects.alt_odb_tail)return ; r->objects.alt_odb_tail = &r->objects.alt_odb_list; -if (!r->ignore_env) { +if (!ros->ignore_env) {const char *alt =getenv (ALTERNATE_DB_ENVIRONMENT );if (!alt) alt ="" ;
-- 8< --
--
Duy
Quoted 14 lines · Duy Nguyen
Quoted 4 lines · Stefan Beller
This is a real take on the first part of the recent RFC[1].
...
Quoted 8 lines
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the
ignore_envflag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.
Quoted 4 lines
Second proposal :) How about you store
ignore_envinraw_object_store? This would not be the first time an object has some configuration passed in at construction time. And it has a "constructor" now,raw_object_store_init()(I probably should merge_setupin it too)
A bit more on this configuration parameters. Down the road I think we need something like this anyway to delete global config vars like packed_git_window_size, delta_base_cache_limit... Either all these end up in raw_object_store, or raw_object_store holds a link to "struct config_set".
The ignore_env specifically though looks to me like a stop gap solution until everything goes through repo_init() first. At that point we don't have to delay getenv() anymore. We can getenv() all at repo_init() then pass them in raw_object_store and ignore_env should be gone. So sticking it inside raw_object_store _temporarily_ does not sound so bad.
Signature · 1 line
Duy
Quoted 26 lines · Duy Nguyen
Quoted 14 lines · Duy Nguyen
Quoted 4 lines · Stefan Beller
This is a real take on the first part of the recent RFC[1].
...
Quoted 8 lines
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the
ignore_envflag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.Quoted 4 lines
Second proposal :) How about you store
ignore_envinraw_object_store? This would not be the first time an object has some configuration passed in at construction time. And it has a "constructor" now,raw_object_store_init()(I probably should merge_setupin it too)A bit more on this configuration parameters. Down the road I think we need something like this anyway to delete global config vars like
packed_git_window_size,delta_base_cache_limit... Either all these end up inraw_object_store, orraw_object_storeholds a link to "structconfig_set".
Quoted 6 lines
The
ignore_envspecifically though looks to me like a stop gap solution until everything goes throughrepo_init()first. At that point we don't have to delaygetenv()anymore. We cangetenv()all atrepo_init()then pass them inraw_object_storeandignore_envshould be gone. So sticking it insideraw_object_store_temporarily_ does not sound so bad.
I like this approach, I mean at the moment we are replicating a single bit of data but that allows us to be able to limit the scope of where a repository struct is passed, giving us a better abstraction layer.
Quoted 2 lines
Signature · 1 line
Duy
Signature · 1 line
Brandon Williams
Quoted 20 lines · Duy Nguyen
Quoted 14 lines · Duy Nguyen
Quoted 4 lines · Stefan Beller
This is a real take on the first part of the recent RFC[1].
...
Quoted 8 lines
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the
ignore_envflag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.Quoted 4 lines
Second proposal :) How about you store
ignore_envinraw_object_store? This would not be the first time an object has some configuration passed in at construction time. And it has a "constructor" now,raw_object_store_init()(I probably should merge_setupin it too)
Quoted 5 lines
A bit more on this configuration parameters. Down the road I think we need something like this anyway to delete global config vars like
packed_git_window_size,delta_base_cache_limit... Either all these end up inraw_object_store, orraw_object_storeholds a link to "structconfig_set".
That makes sense long term.
Quoted 6 lines
The
ignore_envspecifically though looks to me like a stop gap solution until everything goes throughrepo_init()first. At that point we don't have to delaygetenv()anymore. We cangetenv()all atrepo_init()then pass them inraw_object_storeandignore_envshould be gone. So sticking it insideraw_object_store_temporarily_ does not sound so bad.
Oh, that is an interesting perspective. Here is how I arrived at the opposite conclusion initially: Searching for 'ignore_env' shows that we care about it as well for the index and graft paths, which are not the object store, hence it would be better kept in the repository. (The alternative would be to duplicate it into the raw object store, but I do not like data duplication)
But maybe it is better to duplicate this one bit instead of passing through a larger scoped object.
I'll rework the patches.
Thanks!
Stefan
Quoted 8 lines · Stefan Beller
Oh, that is an interesting perspective. Here is how I arrived at the opposite conclusion initially: Searching for '
ignore_env' shows that we care about it as well for the index and graft paths, which are not the object store, hence it would be better kept in the repository. (The alternative would be to duplicate it into the raw object store, but I do not like data duplication)But maybe it is better to duplicate this one bit instead of passing through a larger scoped object.
If a larger scoped repository refers to a smaller scoped object-store, is there still a need to duplicate that bit, instead of referring to the bit the smaller scoped one has when asking about the bit in the larger scoped one?
I am not sure if these "do not look at environment variables" is an attribute of these entities---it sounds more like an attribute for each invocation of an operation, i.e. "I want to learn where the index is but would ignore GIT_INDEX environment for this particular query." and "What's the value of this ref? Please honor the common-dir environment during this query".
So from that point of view, it may not matter where the "bit" lives, among repository, object-store, or ref-store.
Quoted 8 lines · Junio C Hamano
Quoted 6 lines · Stefan Beller
Oh, that is an interesting perspective. Here is how I arrived at the opposite conclusion initially: Searching for '
ignore_env' shows that we care about it as well for the index and graft paths, which are not the object store, hence it would be better kept in the repository. (The alternative would be to duplicate it into the raw object store, but I do not like data duplication)
Quoted 7 lines
Quoted 2 lines
But maybe it is better to duplicate this one bit instead of passing through a larger scoped object.
If a larger scoped repository refers to a smaller scoped object-store, is there still a need to duplicate that bit, instead of referring to the bit the smaller scoped one has when asking about the bit in the larger scoped one?
No (in theory). But in practice it may be worthwhile:
"What's the value of this ref?"
"Oh let me check the ignore_env bit that happens to live in the object store first."
would be super confusing to me.
Quoted 6 lines
I am not sure if these "do not look at environment variables" is an attribute of these entities---it sounds more like an attribute for each invocation of an operation, i.e. "I want to learn where the index is but would ignore
GIT_INDEXenvironment for this particular query." and "What's the value of this ref? Please honor the common-dir environment during this query".
That sounds like we want to have a configset struct eventually.
For now the ignore_env bit lives in the repository, as that helps when working with submodules, when reading its comments. Unfortunately 359efeffc1 (repository: introduce the repository object, 2017-06-22) did not reason about the existence of the ignore_env flag in its commit message.
Quoted 2 lines
So from that point of view, it may not matter where the "bit" lives, among repository, object-store, or ref-store.
It matters on the scale of confusing the developer?
Stefan
Quoted 10 lines · Stefan Beller
For now the
ignore_envbit lives in the repository, as that helps when working with submodules, when reading its comments. Unfortunately359efeffc1(repository: introduce the repository object, 2017-06-22) did not reason about the existence of theignore_envflag in its commit message.Quoted 2 lines
So from that point of view, it may not matter where the "bit" lives, among repository, object-store, or ref-store.
It matters on the scale of confusing the developer?
What I meant is that from the point of view, having the bit in the data (not on the invocation) is confusing no matter which data structure holds it--they are equally bad.
Quoted 8 lines · Junio C Hamano
Quoted 6 lines · Stefan Beller
For now the
ignore_envbit lives in the repository, as that helps when working with submodules, when reading its comments. Unfortunately359efeffc1(repository: introduce the repository object, 2017-06-22) did not reason about the existence of theignore_envflag in its commit message.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
So from that point of view, it may not matter where the "bit" lives, among repository, object-store, or ref-store.
It matters on the scale of confusing the developer?
What I meant is that from the point of view, having the bit in the data (not on the invocation) is confusing no matter which data structure holds it--they are equally bad.
Right.
Which is why I'd strongly consider having it only in the repository object as that is the largest-scoped thing we'd want. e.g. submodules should care about environment variables differently:
GIT_WORK_TREE=~/mysuperproject git checkout \
--recurse-submodules master So with such a command in mind, the environment variable would
only apply to the superproject and the nested submodules should ignore the env, but compute their paths off the superproject, i.e. the superproject repository, not its object store?
Quoted 11 lines · Stefan Beller
Which is why I'd strongly consider having it only in the repository object as that is the largest-scoped thing we'd want. e.g. submodules should care about environment variables differently:
GIT_WORK_TREE=~/mysuperproject git checkout \ --recurse-submodules masterSo with such a command in mind, the environment variable would
only apply to the superproject and the nested submodules should ignore the env, but compute their paths off the superproject, i.e. the superproject repository, not its object store?
In the longer term endgame state, what is in the repository object cannot just be a simple "do we honor environment variable" bit anymore.
It will be more like "we may (or may not) look at environment when we create a repository object", i.e. a bit passed to repo_init() constructor, and from then on, the repository object knows where the object store and its alternates are, where the top of the working tree is, where the repository (i.e. the directory that has "refs/" in it) is, what "worktree" of the repository we are talking about by itself. There is no need for a bit "do we or do we not look at environment?" that needs to be consulted at runtime, which is quite a round-about thing.
In your example, the repository object that represents the superproject will pay attention to GIT_WORK_TREE environment when it is consturcted, and repository objects dynamically constructed while the command "recurse-submodules" through them will be constructed with their "where is the top of my working tree?" set appropriately. They won't be storing "when figuring out where my working tree is, do not pay attention to GIT_WORK_TREE environment variable" bit.
Hi,
Quoted 6 lines · Stefan Beller
This is a real take on the first part of the recent RFC[1].
Jonathan Tan suggested[2] that "
sha1_loose_object_infoto handle arbitrary repositories" might be a good breaking point for a first part at that RFC at patch 38. This series is smaller and contains only 26 patches as the patches in the big RFC were slightly out of order.
Thanks. This looks like a nice reviewable series, so I'm happy to see it broken out.
[...]
Quoted 3 lines
Comments in the early range of that RFC were on 003 where Junio pointed out that the coccinelle patch ought to be not in contrib/coccinelle, so I put it in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.
Can you say a little more about this? Was the problem that the semantic patch wasn't idempotent, that it was too slow to run, or something else?
If we're including the semantic patch for reference but never running it, I think I'd prefer it to go in the commit message. But if it's useful to run then we should make it idempotent so it can go in contrib/coccinelle.
[...]
Quoted 6 lines
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the
ignore_envflag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.
I think I want to push back on this a little.
The advantage of a function taking e.g. an object_store as an argument instead of a repository is that it increases its flexibility, since it allows callers that do not have access to a repository to call it. The disadvantage is also that it increases the flexibility without any callers benefitting from that:
1. It ties us to assumptions from today. If e.g. an object access in the future starts relying on some other information from the repository (e.g. its config) then we'd have to either add a back-pointer from the object store to its repository or add additional arguments for that additional data at that point.
If all callers already have a repository, it is simpler to pass that repository as context so that we have the flexibility to make more use of it later.
2. It complicates the caller. Instead of consistently passing the same repository argument as context to functions that access that repository, the caller would have to pull out relevant fields like the object store from it.
3. It prevents us from making opportunistic use of other information from the repository, such as its name for use in error messages.
In lower-level funcitons that need to be usable by callers without a repository (e.g. to find packfiles in an alternate) it makes sense to not pass a repository, but without such a use case in mind I don't think it needs to be a general goal.
To put it another way, most callers do not *care* whether they are working with a repository's object store, ref database, or some other aspect of the repository. They just know they want to e.g. read an object from this repository.
It's similar to how FILE * works: some operations rely on the buffer the FILE * manages and some other operations only rely on the underlying file descriptor, but using the FILE * consistently provides a clean abstraction that generally makes life easier.
Quoted 2 lines
Eric Sunshine brought up memory leaks with the RFC, and I would think to have plugged all holes.
Yay, thank you!
I'll try to find time to look at the patches in detail soon, but no promises (i.e. if someone else reviews them first, then even better ;-)).
Sincerely,
Jonathan
Quoted 21 lines · Jonathan Nieder
Quoted 6 lines
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the
ignore_envflag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.I think I want to push back on this a little.
The advantage of a function taking e.g. an
object_storeas an argument instead of a repository is that it increases its flexibility, since it allows callers that do not have access to a repository to call it. The disadvantage is also that it increases the flexibility without any callers benefitting from that:1. It ties us to assumptions from today. If e.g. an object access in the future starts relying on some other information from the repository (e.g. its config) then we'd have to either add a back-pointer from the object store to its repository or add additional arguments for that additional data at that point.
Quoted 3 lines
If all callers already have a repository, it is simpler to pass that repository as context so that we have the flexibility to make more use of it later.
It's essentially putting all global variables in the same place again. Only this time it's not global namespace, but "struct repository". It's no worse than the current state though.
Quoted 5 lines
2. It complicates the caller. Instead of consistently passing the same repository argument as context to functions that access that repository, the caller would have to pull out relevant fields like the object store from it.
Well, I see that as a good point :)
Quoted 3 lines
3. It prevents us from making opportunistic use of other information from the repository, such as its name for use in error messages.
It does not exactly prevent us. It's just more effort to pass this around. The repository name for example, there's no reason we can't have object store name (which could be initialized the same as repo name).
Quoted 3 lines
In lower-level funcitons that need to be usable by callers without a repository (e.g. to find packfiles in an alternate) it makes sense to not pass a repository, but without such a use case in mind
I do agree with your benefit argument. But I'd like to point out though that having all input to object store visible from something like "struct raw_object_store" makes it easier to reason about the code. I know how object store works, but occasionally I'm still surprised when it getenv (or read $GIT_DIR/index, but not in object store code) behind the scene. Imagine how hard it is for newcomers.
I would count that as benefit, even though it's not a use case per se. Another potential benefit is writing unit tests will be much easier (you can configure object store through struct repository too, but setting one piece here, one piece there to control object store behavior is not a nice experience). It's a nice thing to have, but not a deciding factor.
Quoted 1 line
I don't think it needs to be a general goal.
My stand is a bit more aggressive. We should try to achieve better abstraction if possible. But if it makes Stefan's life hell, it's not worth doing. Converting to 'struct repository' is already a step forward. Actually if it discourages him from finishing this work, it's already not worth doing.
Signature · 1 line
Duy
Quoted 25 lines · Stefan Beller
This is a real take on the first part of the recent RFC[1].
Jonathan Tan suggested[2] that "
sha1_loose_object_infoto handle arbitrary repositories" might be a good breaking point for a first part at that RFC at patch 38. This series is smaller and contains only 26 patches as the patches in the big RFC were slightly out of order.I developed this series partly by writing patches, but mostly by cherrypicking from that RFC on top of current master. I noticed no external conflicts apart from one addition to the repositories
_INITmacro, which was easy to resolve.Comments in the early range of that RFC were on 003 where Junio pointed out that the coccinelle patch ought to be not in contrib/coccinelle, so I put it in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.
brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself without any suggestion to include into this series[3].
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the
ignore_envflag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.
Quoted 2 lines
Eric Sunshine brought up memory leaks with the RFC, and I would think to have plugged all holes.
I've looked through the patches and I think they look good. At the end of the series all the #define tricks have been eliminated so we don't have to worry about them possibly being left and forgotten :)
Thanks for getting the ball rolling on this.
Quoted 83 lines
[1] https://public-inbox.org/git/[email protected]/
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/Thanks,
StefanJonathan Nieder (8):
pack: moveprepare_packed_git_run_onceto object store
pack: move approximate object count to object store
sha1_file: add repository argument tosha1_file_name
sha1_file: add repository argument tomap_sha1_file
sha1_file: allowstat_sha1_fileto handle arbitrary repositories
sha1_file: allowopen_sha1_fileto handle arbitrary repositories
sha1_file: allowmap_sha1_file_1to handle arbitrary repositories
sha1_file: allowsha1_loose_object_infoto handle arbitrary
repositoriesStefan Beller (18):
repository: introduce raw object store field
object-store: movealt_odb_listandalt_odb_tailto object store
object-store: freealt_odb_list
object-store: movepacked_gitandpacked_git_mruto object store
object-store: close all packs upon clearing the object store
sha1_file: addraw_object_storeargument toalt_odb_usable
sha1_file: add repository argument tolink_alt_odb_entry
sha1_file: add repository argument toread_info_alternates
sha1_file: add repository argument tolink_alt_odb_entries
sha1_file: add repository argument toprepare_alt_odb
sha1_file: allowlink_alt_odb_entriesto handle arbitrary repositories
sha1_file: allowprepare_alt_odbto handle arbitrary repositories
sha1_file: add repository argument tostat_sha1_file
sha1_file: add repository argument toopen_sha1_file
sha1_file: add repository argument tomap_sha1_file_1
sha1_file: add repository argument tosha1_loose_object_info
sha1_file: allowsha1_file_nameto handle arbitrary repositories
sha1_file: allowmap_sha1_fileto handle arbitrary repositoriesbuiltin/am.c | 2 +- builtin/clone.c | 2 +- builtin/count-objects.c | 6 +- builtin/fetch.c | 2 +- builtin/fsck.c | 13 ++- builtin/gc.c | 4 +- builtin/grep.c | 2 +- builtin/index-pack.c | 1 + builtin/merge.c | 2 +- builtin/pack-objects.c | 21 ++-- builtin/pack-redundant.c | 6 +- builtin/receive-pack.c | 3 +- cache.h | 46 ++------ contrib/coccinelle/refactoring/packed_git.cocci | 7 ++ environment.c | 5 +- fast-import.c | 6 +- http-backend.c | 6 +- http-push.c | 1 + http-walker.c | 4 +- http.c | 6 +- mru.h | 1 + object-store.h | 75 +++++++++++++ object.c | 26 +++++ pack-bitmap.c | 4 +- pack-check.c | 1 + pack-revindex.c | 1 + packfile.c | 64 +++++------ packfile.h | 2 +- path.c | 2 +- reachable.c | 1 + repository.c | 17 ++- repository.h | 7 +- server-info.c | 6 +- sha1_file.c | 135 +++++++++++++----------- sha1_name.c | 11 +- streaming.c | 5 +- 36 files changed, 321 insertions(+), 182 deletions(-) create mode 100644 contrib/coccinelle/refactoring/packed_git.cocci create mode 100644 object-store.hSignature · 1 line
2.16.1.73.ga2c3e9663f.dirty
Signature · 1 line
Brandon Williams
Quoted 1 line · Stefan Beller
This is a real take on the first part of the recent RFC[1].
For the patches remaining in this series, The scope is about right and the size is more manageable. With topics already on 'master', they have some interactions:
- ot/mru-on-list & gs/retire-mru
The packed_git MRU has been greatly simplified by using list API directly.
- cc/sha1-file-name
The function signature of sha1_file_name() has been updated.
I could certainly carry evil merge resolutions going forward (these changes need to be made to object-store.h that has no mechanical conflicts), but it may be less distracting for later readers of the series if we rebase it on top of a more recent master.
Quoted 6 lines · Junio C Hamano
Quoted 1 line · Stefan Beller
This is a real take on the first part of the recent RFC[1].
For the patches remaining in this series, The scope is about right and the size is more manageable.
ok, thanks.
Quoted 8 lines
With topics already on 'master',
they have some interactions:- ot/mru-on-list & gs/retire-mru
The
packed_gitMRU has been greatly simplified by using list API directly.
Quoted 8 lines
- cc/sha1-file-name
The function signature of
sha1_file_name()has been updated.I could certainly carry evil merge resolutions going forward (these changes need to be made to
object-store.hthat has no mechanical conflicts), but it may be less distracting for later readers of the series if we rebase it on top of a more recent master.
I was debating when to reroll this series as I want to make the change that Duy proposed, moving the 'ignore_env' into the object store as well.
I'll rebase on top of the latest master or these topicc while at it.
Thanks for the heads up.
Stefan
v2 cover Moving global state into the repository object (part 1)
In reply to Junio C Hamanov2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global, which means there is just one patch per function that is converted. As this follows the same structure of the previous series, I am still confident there is no hidden dependencies to globals outside the object store in these converted functions.
* rebased on top of current master, resolving the merge conflicts.
I think I used the list.h APIs right, but please double check.
Thanks,
Stefan
v1:
This is a real take on the first part of the recent RFC[1].
Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories" might be a good breaking point for a first part at that RFC at patch 38. This series is smaller and contains only 26 patches as the patches in the big RFC were slightly out of order.
I developed this series partly by writing patches, but mostly by cherrypicking from that RFC on top of current master. I noticed no external conflicts apart from one addition to the repositories _INIT macro, which was easy to resolve.
Comments in the early range of that RFC were on 003 where Junio pointed out that the coccinelle patch ought to be not in contrib/coccinelle, so I put it in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.
brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself without any suggestion to include into this series[3].
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the ignore_env flag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.
Eric Sunshine brought up memory leaks with the RFC, and I would think to have plugged all holes.
[1] https://public-inbox.org/git/[email protected]/
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/
Thanks,
Stefan
Stefan Beller (16):
repository: introduce raw object store field
object-store: move alt_odb_list and alt_odb_tail to object store
object-store: free alt_odb_list
object-store: move packed_git and packed_git_mru to object store
object-store: close all packs upon clearing the object store
pack: move prepare_packed_git_run_once to object store
pack: move approximate object count to object store
sha1_file: add raw_object_store argument to alt_odb_usable
sha1_file: allow link_alt_odb_entries to handle arbitrary object
stores
sha1_file: allow prepare_alt_odb to handle arbitrary object stores
sha1_file: allow sha1_file_name to handle arbitrary object stores
sha1_file: allow stat_sha1_file to handle arbitrary object stores
sha1_file: allow open_sha1_file to handle arbitrary object stores
sha1_file: allow map_sha1_file_1 to handle arbitrary object stores
sha1_file: allow map_sha1_file to handle arbitrary object stores
sha1_file: allow sha1_loose_object_info to handle arbitrary object
stores
builtin/am.c | 2 +- builtin/clone.c | 2 +- builtin/count-objects.c | 6 +- builtin/fetch.c | 2 +- builtin/fsck.c | 13 +++-- builtin/gc.c | 4 +- builtin/grep.c | 2 +- builtin/index-pack.c | 1 + builtin/merge.c | 2 +- builtin/pack-objects.c | 19 +++--- builtin/pack-redundant.c | 6 +- builtin/receive-pack.c | 3 +- cache.h | 45 ++------------ environment.c | 5 +- fast-import.c | 6 +- http-backend.c | 6 +- http-push.c | 1 + http-walker.c | 4 +- http.c | 6 +- object-store.h | 80 +++++++++++++++++++++++++ object.c | 27 +++++++++ pack-bitmap.c | 4 +- pack-check.c | 1 + pack-revindex.c | 1 + packfile.c | 64 ++++++++++---------- packfile.h | 2 +- path.c | 2 +- reachable.c | 1 + repository.c | 22 +++++-- repository.h | 7 ++- server-info.c | 6 +- sha1_file.c | 123 +++++++++++++++++++++------------------ sha1_name.c | 11 ++-- streaming.c | 5 +- 34 files changed, 314 insertions(+), 177 deletions(-) create mode 100644 object-store.h
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 1/16 repository: introduce raw object store field
The raw object store field will contain any objects needed for access to objects in a given repository.
This patch introduces the raw object store and populates it with the objectdir, which used to be part of the repository struct as well as 'ignore_env' which is duplicated from the repository. A later refactoring (not in this series) will hopefully get rid of that one bit, once all the construction of repo objects goes through repo_init(). The reason for duplication now is that the refactoring done in this series allows for smaller scoped objects in the functions refactored, nameny passing around the object store instead of the repository object.
As the struct gains members, we'll also populate the function to clear the memory for these members.
In a later we'll introduce a struct object_parser, that will complement the object parsing in a repository struct: The raw object parser is the layer that will provide access to raw object content, while the higher level object parser code will parse raw objects and keeps track of parenthood and other object relationships using 'struct object'. For now only add the lower level to the repository struct.
--- builtin/grep.c | 2 +- environment.c | 5 +++-- object-store.h | 17 +++++++++++++++++ object.c | 5 +++++ path.c | 2 +- repository.c | 19 +++++++++++++++---- repository.h | 7 ++++--- 7 files changed, 46 insertions(+), 11 deletions(-) create mode 100644 object-store.h
builtin/grep.c +1 -1
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,* object. */ grep_read_lock (); -add_to_alternates_memory (submodule.objectdir); +add_to_alternates_memory (submodule.objects.objectdir);grep_read_unlock ();if (oid) {
environment.c +3 -2
@@ -13,6 +13,7 @@#include "refs.h" #include "fmt-merge-msg.h" #include "commit.h" +#include "object-store.h" int trust_executable_bit =1 ;int trust_ctime =1 ;
@@ -244,9 +245,9 @@ const char *get_git_work_tree(void)char *get_object_directory (void ) { -if (!the_repository->objectdir) +if (!the_repository->objects.objectdir)BUG ("git environment hasn't been setup" ); -return the_repository->objectdir; +return the_repository->objects.objectdir; }int odb_mkstemp (struct strbuf *template,const char *pattern)
object-store.h new file +17 -0
@@ -0,0 +1,17 @@ +#ifndef OBJECT_STORE_H +#define OBJECT_STORE_H + +struct raw_object_store { +/* +* Path to the repository's object store. +* Cannot be NULL after initialization. +*/ +char *objectdir; + +unsigned ignore_env :1 ; +}; +#define RAW_OBJECT_STORE_INIT { NULL, 0 } + +void raw_object_store_clear (struct raw_object_store *o); + +#endif /* OBJECT_STORE_H */
object.c +5 -0
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags) obj->flags &= ~flags; } } + +void raw_object_store_clear (struct raw_object_store *o) +{ +free (o->objectdir); +}
path.c +1 -1
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,strbuf_splice (buf,0 , buf->len, repo->index_file,strlen (repo->index_file)); else if (dir_prefix(base, "objects")) - replace_dir(buf, git_dir_len + 7, repo->objectdir); + replace_dir(buf, git_dir_len + 7, repo->objects.objectdir); else if (git_hooks_path && dir_prefix(base, "hooks")) replace_dir(buf, git_dir_len + 5, git_hooks_path); else if (repo->different_commondir)
repository.c +15 -4
@@ -1,11 +1,18 @@#include "cache.h" #include "repository.h" +#include "object-store.h" #include "config.h" #include "submodule-config.h" /* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL , +RAW_OBJECT_STORE_INIT , +NULL ,NULL ,NULL , +NULL ,NULL ,NULL , + &the_index, + &hash_algos[GIT_HASH_SHA1 ], +0 ,0 };struct repository *the_repository = &the_repo;
@@ -42,9 +49,11 @@ static void repo_setup_env(struct repository *repo) !repo->ignore_env);free (repo->commondir); repo->commondir =strbuf_detach (&sb,NULL ); -free (repo->objectdir); - repo->objectdir =git_path_from_env (DB_ENVIRONMENT , repo->commondir, +free (repo->objects.objectdir); + repo->objects.objectdir =git_path_from_env (DB_ENVIRONMENT , repo->commondir,"objects" , !repo->ignore_env); + repo->objects.ignore_env = repo->ignore_env; +free (repo->graft_file); repo->graft_file =git_path_from_env (GRAFT_ENVIRONMENT , repo->commondir,"info/grafts" , !repo->ignore_env);
@@ -209,12 +218,14 @@ void repo_clear(struct repository *repo) {FREE_AND_NULL (repo->gitdir);FREE_AND_NULL (repo->commondir); -FREE_AND_NULL (repo->objectdir);FREE_AND_NULL (repo->graft_file);FREE_AND_NULL (repo->index_file);FREE_AND_NULL (repo->worktree);FREE_AND_NULL (repo->submodule_prefix); +raw_object_store_clear (&repo->objects); +memset (&repo->objects,0 ,sizeof (repo->objects)); +if (repo->config) {git_configset_clear (repo->config);FREE_AND_NULL (repo->config);
repository.h +4 -3
@@ -1,6 +1,8 @@#ifndef REPOSITORY_H #define REPOSITORY_H +#include "object-store.h" +struct config_set ;struct index_state ;struct submodule_cache ;
@@ -21,10 +23,9 @@ struct repository {char *commondir;/* -* Path to the repository's object store. -* Cannot be NULL after initialization. +* Holds any information related to the object store. */ -char *objectdir; +struct raw_object_store objects;/* * Path to the repository's graft file.
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 2/16 object-store: move alt_odb_list and alt_odb_tail to object store
In reply to Stefan BellerIn a process with multiple repositories open, alternates should be associated to a single repository and not shared globally. Move alt_odb_list and alt_odb_tail into the_repository and adjust callers to reflect this.
Now that the alternative object data base is per repository, we're leaking its memory upon freeing a repository. The next patch plugs this hole.
No functional change intended.
--- builtin/fsck.c | 4 +++- cache.h | 4 ++-- object-store.h | 7 ++++++- packfile.c | 3 ++- sha1_file.c | 25 ++++++++++++------------- sha1_name.c | 3 ++- 6 files changed, 27 insertions(+), 19 deletions(-)
builtin/fsck.c +3 -1
@@ -1,5 +1,6 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "commit.h" #include "tree.h"
@@ -716,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)fsck_object_dir (get_object_directory ());prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; + alt; alt = alt->next)fsck_object_dir (alt->path);if (check_full) {
cache.h +2 -2
@@ -1576,7 +1576,7 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);extern void schedule_dir_for_removal (const char *name,int len);extern void remove_scheduled_dirs (void ); -extern struct alternate_object_database { +struct alternate_object_database {struct alternate_object_database *next;/* see alt_scratch_buf() */
@@ -1594,7 +1594,7 @@ extern struct alternate_object_database {struct oid_array loose_objects_cache;char path[FLEX_ARRAY ]; -} *alt_odb_list; +};extern void prepare_alt_odb (void );extern char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);
object-store.h +6 -1
@@ -1,6 +1,8 @@#ifndef OBJECT_STORE_H #define OBJECT_STORE_H +#include "cache.h" + struct raw_object_store {/* * Path to the repository's object store.
@@ -8,9 +10,12 @@ struct raw_object_store {*/ char *objectdir; +struct alternate_object_database *alt_odb_list; +struct alternate_object_database **alt_odb_tail; + unsigned ignore_env : 1; }; -#define RAW_OBJECT_STORE_INIT { NULL, 0 } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL, 0 }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +2 -1
@@ -1,6 +1,7 @@#include "cache.h" #include "list.h" #include "pack.h" +#include "repository.h" #include "dir.h" #include "mergesort.h" #include "packfile.h"
@@ -891,7 +892,7 @@ void prepare_packed_git(void)return ;prepare_packed_git_one (get_object_directory (),1 );prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();prepare_packed_git_mru ();
sha1_file.c +12 -13
@@ -22,6 +22,7 @@#include "pack-revindex.h" #include "sha1-lookup.h" #include "bulk-checkin.h" +#include "repository.h" #include "streaming.h" #include "dir.h" #include "list.h"
@@ -342,9 +343,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,return buf->buf; } -struct alternate_object_database *alt_odb_list; -static struct alternate_object_database **alt_odb_tail; -/* * Return non-zero iff the path is usable as an alternate object database. */
@@ -364,7 +362,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)* Prevent the common mistake of listing the same * thing twice, or object directory itself. */ -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {if (!fspathcmp (path->buf, alt->path))return 0 ; }
@@ -424,8 +422,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base, ent =alloc_alt_odb (pathbuf.buf);/* add the alternate entry */ - *alt_odb_tail = ent; - alt_odb_tail = &(ent->next); + *the_repository->objects.alt_odb_tail = ent; + the_repository->objects.alt_odb_tail = &(ent->next); ent->next =NULL ;/* recursively add alternates */
@@ -559,7 +557,7 @@ void add_to_alternates_file(const char *reference)fprintf_or_die (out,"%s\n" , reference);if (commit_lock_file (&lock))die_errno ("unable to move new alternates file into place" ); -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)link_alt_odb_entries (reference,'\n' ,NULL ,0 ); }free (alts);
@@ -657,7 +655,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)int r =0 ;prepare_alt_odb (); -for (ent = alt_odb_list; ent; ent = ent->next) { +for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) { r =fn (ent, cb);if (r)break ;
@@ -669,12 +667,13 @@ void prepare_alt_odb(void) {const char *alt; -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)return ; alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - alt_odb_tail = &alt_odb_list; + the_repository->objects.alt_odb_tail = + &the_repository->objects.alt_odb_list;link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 );read_info_alternates (get_object_directory (),0 );
@@ -718,7 +717,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen) {struct alternate_object_database *alt;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {const char *path =alt_sha1_path (alt, sha1);if (check_and_freshen_file (path, freshen))return 1 ;
@@ -878,7 +877,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,prepare_alt_odb (); errno =ENOENT ; -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);if (!lstat (*path, st))return 0 ;
@@ -908,7 +907,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path) most_interesting_errno = errno;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);if (fd >=0 )
sha1_name.c +2 -1
@@ -10,6 +10,7 @@#include "dir.h" #include "sha1-array.h" #include "packfile.h" +#include "repository.h" static int get_oid_oneline (const char *,struct object_id *,struct commit_list *);
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)*/ fakeent =alloc_alt_odb (get_object_directory ()); } - fakeent->next = alt_odb_list; + fakeent->next = the_repository->objects.alt_odb_list;for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {int pos;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 3/16 object-store: free alt_odb_list
In reply to Stefan BellerFree the memory and reset alt_odb_{list, tail} to NULL.
--- object.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
object.c +17 -0
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags) } } +static void free_alt_odb (struct alternate_object_database *alt) +{ +strbuf_release (&alt->scratch); +oid_array_clear (&alt->loose_objects_cache); +} + +static void free_alt_odbs (struct raw_object_store *o) +{ +while (o->alt_odb_list) { +free_alt_odb (o->alt_odb_list); + o->alt_odb_list = o->alt_odb_list->next; + } +} +void raw_object_store_clear (struct raw_object_store *o) {free (o->objectdir); + +free_alt_odbs (o); + o->alt_odb_tail =NULL ; }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 4/16 object-store: move packed_git and packed_git_mru to object store
In reply to Stefan BellerIn a process with multiple repositories open, packfile accessors should be associated to a single repository and not shared globally. Move packed_git and packed_git_mru into the_repository and adjust callers to reflect this.
Patch generated by
1. Moving the struct packed_git declaration to object-store.h and packed_git, packed_git_mru globals to struct object_store.
2. Applying the semantic patch to adjust callers. @@ @@
- packed_git
+ the_repository->objects.packed_git
@@ @@
- packed_git_mru
+ the_repository->objects.packed_git_mru
3. Applying line wrapping fixes from "make style" to break the resulting long lines.
4. Adding missing #includes of repository.h and object-store.h where needed.
5. As the packfiles are now owned by an objectstore/repository, which is ephemeral unlike globals, we introduce memory leaks. So address them in raw_object_store_clear().
--- builtin/count-objects.c | 6 ++++-- builtin/fsck.c | 7 +++++-- builtin/gc.c | 4 +++- builtin/index-pack.c | 1 + builtin/pack-objects.c | 19 +++++++++++-------- builtin/pack-redundant.c | 6 ++++-- builtin/receive-pack.c | 1 + cache.h | 29 ----------------------------- fast-import.c | 6 ++++-- http-backend.c | 6 ++++-- http-push.c | 1 + http-walker.c | 1 + http.c | 1 + object-store.h | 36 +++++++++++++++++++++++++++++++++++- object.c | 7 +++++++ pack-bitmap.c | 4 +++- pack-check.c | 1 + pack-revindex.c | 1 + packfile.c | 39 ++++++++++++++++++++------------------- reachable.c | 1 + repository.c | 3 +++ server-info.c | 6 ++++-- sha1_name.c | 5 +++-- 23 files changed, 118 insertions(+), 73 deletions(-)
builtin/count-objects.c +4 -2
@@ -7,6 +7,8 @@#include "cache.h" #include "config.h" #include "dir.h" +#include "repository.h" +#include "object-store.h" #include "builtin.h" #include "parse-options.h" #include "quote.h"
@@ -120,9 +122,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)struct strbuf loose_buf =STRBUF_INIT ;struct strbuf pack_buf =STRBUF_INIT ;struct strbuf garbage_buf =STRBUF_INIT ; -if (!packed_git) +if (!the_repository->objects.packed_git)prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ;if (open_pack_index (p))
builtin/fsck.c +5 -2
@@ -2,6 +2,7 @@#include "cache.h" #include "repository.h" #include "config.h" +#include "object-store.h" #include "commit.h" #include "tree.h" #include "blob.h"
@@ -729,7 +730,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)prepare_packed_git ();if (show_progress) { -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {if (open_pack_index (p))continue ; total += p->num_objects;
@@ -737,7 +739,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) progress =start_progress (_ ("Checking objects" ), total); } -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {/* verify gives error messages itself */ if (verify_pack (p, fsck_obj_buffer, progress, count))
builtin/gc.c +3 -1
@@ -11,6 +11,7 @@*/ #include "builtin.h" +#include "repository.h" #include "config.h" #include "tempfile.h" #include "lockfile.h"
@@ -18,6 +19,7 @@#include "run-command.h" #include "sigchain.h" #include "argv-array.h" +#include "object-store.h" #include "commit.h" #include "packfile.h"
@@ -173,7 +175,7 @@ static int too_many_packs(void)return 0 ;prepare_packed_git (); -for (cnt =0 , p = packed_git; p; p = p->next) { +for (cnt =0 , p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ;if (p->pack_keep)
builtin/index-pack.c +1 -0
@@ -12,6 +12,7 @@#include "exec_cmd.h" #include "streaming.h" #include "thread-utils.h" +#include "object-store.h" #include "packfile.h" static const char index_pack_usage[] =
builtin/pack-objects.c +11 -8
@@ -1,6 +1,8 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" +#include "object-store.h" #include "attr.h" #include "object.h" #include "blob.h"
@@ -1026,7 +1028,7 @@ static int want_object_in_pack(const struct object_id *oid,return want; } -list_for_each (pos, &packed_git_mru) { +list_for_each (pos, &the_repository->objects.packed_git_mru) { struct packed_git *p = list_entry(pos, struct packed_git, mru);off_t offset;
@@ -1044,7 +1046,7 @@ static int want_object_in_pack(const struct object_id *oid, } want =want_found_object (exclude, p);if (!exclude && want >0 ) -list_move (&p->mru, &packed_git_mru); +list_move (&p->mru, &the_repository->objects.packed_git_mru);if (want !=-1 )return want; }
@@ -2673,7 +2675,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)memset (&in_pack,0 ,sizeof (in_pack)); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {struct object_id oid;struct object *o;
@@ -2736,7 +2738,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)static struct packed_git *last_found = (void *)1 ;struct packed_git *p; - p = (last_found != (void *)1 ) ? last_found : packed_git; + p = (last_found != (void *)1 ) ? last_found : + the_repository->objects.packed_git;while (p) { if ((!p->pack_local || p->pack_keep) &&
@@ -2745,7 +2748,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)return 1 ; }if (p == last_found) - p = packed_git; + p = the_repository->objects.packed_git;else p = p->next;if (p == last_found)
@@ -2781,7 +2784,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)uint32_t i;struct object_id oid; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local || p->pack_keep)continue ;
@@ -3152,7 +3155,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)prepare_packed_git ();if (ignore_packed_keep) {struct packed_git *p; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)if (p->pack_local && p->pack_keep)break ;if (!p)/* no keep-able packs found */
@@ -3165,7 +3168,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)* also covers non-local objects */ struct packed_git *p; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local) { have_non_local_packs =1 ;break ;
builtin/pack-redundant.c +4 -2
@@ -7,6 +7,8 @@*/ #include "builtin.h" +#include "repository.h" +#include "object-store.h" #include "packfile.h" #define BLKSIZE 512
@@ -571,7 +573,7 @@ static struct pack_list * add_pack(struct packed_git *p)static struct pack_list *add_pack_file (const char *filename) { -struct packed_git *p = packed_git; +struct packed_git *p = the_repository->objects.packed_git;if (strlen (filename) <40 )die ("Bad pack filename: %s" , filename);
@@ -586,7 +588,7 @@ static struct pack_list * add_pack_file(const char *filename)static void load_all (void ) { -struct packed_git *p = packed_git; +struct packed_git *p = the_repository->objects.packed_git;while (p) {add_pack (p);
builtin/receive-pack.c +1 -0
@@ -7,6 +7,7 @@#include "sideband.h" #include "run-command.h" #include "exec_cmd.h" +#include "object-store.h" #include "commit.h" #include "object.h" #include "remote.h"
cache.h +0 -29
@@ -1636,35 +1636,6 @@ struct pack_window {unsigned int inuse_cnt; }; -extern struct packed_git { -struct packed_git *next; -struct list_head mru; -struct pack_window *windows; -off_t pack_size; -const void *index_data; -size_t index_size; -uint32_t num_objects; -uint32_t num_bad_objects; -unsigned char *bad_object_sha1; -int index_version; -time_t mtime; -int pack_fd; -unsigned pack_local:1 , - pack_keep:1 , - freshened:1 , - do_not_close:1 , - pack_promisor:1 ; -unsigned char sha1[20 ]; -struct revindex_entry *revindex; -/* something like ".git/objects/pack/xxxxx.pack" */ -char pack_name[FLEX_ARRAY ];/* more */ -} *packed_git; - -/* -* A most-recently-used ordered version of the packed_git list. -*/ -extern struct list_head packed_git_mru; -struct pack_entry {off_t offset;unsigned char sha1[20 ];
fast-import.c +4 -2
@@ -154,8 +154,10 @@ Format of STDIN stream:#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "lockfile.h" +#include "object-store.h" #include "object.h" #include "blob.h" #include "tree.h"
@@ -1110,7 +1112,7 @@ static int store_object(if (e->idx.offset) { duplicate_count_by_type[type]++;return 1 ; - }else if (find_sha1_pack (oid.hash, packed_git)) { + }else if (find_sha1_pack (oid.hash, the_repository->objects.packed_git)) { e->type = type; e->pack_id =MAX_PACK_ID ; e->idx.offset =1 ;/* just not zero! */
@@ -1305,7 +1307,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark) duplicate_count_by_type[OBJ_BLOB ]++;truncate_pack (&checkpoint); - }else if (find_sha1_pack (oid.hash, packed_git)) { + }else if (find_sha1_pack (oid.hash, the_repository->objects.packed_git)) { e->type =OBJ_BLOB ; e->pack_id =MAX_PACK_ID ; e->idx.offset =1 ;/* just not zero! */
http-backend.c +4 -2
@@ -1,7 +1,9 @@#include "cache.h" #include "config.h" +#include "repository.h" #include "refs.h" #include "pkt-line.h" +#include "object-store.h" #include "object.h" #include "tag.h" #include "exec_cmd.h"
@@ -518,13 +520,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)select_getanyfile (hdr);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_local) cnt++; }strbuf_grow (&buf, cnt *53 +2 ); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_local)strbuf_addf (&buf,"P %s\n" , p->pack_name + objdirlen +6 ); }
http-push.c +1 -0
@@ -1,4 +1,5 @@#include "cache.h" +#include "object-store.h" #include "commit.h" #include "tag.h" #include "blob.h"
http-walker.c +1 -0
@@ -4,6 +4,7 @@#include "http.h" #include "list.h" #include "transport.h" +#include "object-store.h" #include "packfile.h" struct alt_base {
http.c +1 -0
@@ -1,6 +1,7 @@#include "git-compat-util.h" #include "http.h" #include "config.h" +#include "object-store.h" #include "pack.h" #include "sideband.h" #include "run-command.h"
object-store.h +35 -1
@@ -2,6 +2,7 @@#define OBJECT_STORE_H #include "cache.h" +#include "list.h" struct raw_object_store {/*
@@ -10,13 +11,46 @@ struct raw_object_store {*/ char *objectdir; +struct packed_git *packed_git; +/* +* A most-recently-used ordered version of the packed_git list, which can +* be iterated instead of packed_git (and marked via mru_mark). +*/ +struct list_head packed_git_mru; +struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; unsigned ignore_env : 1; }; -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL, 0 } + +#define MRU_LIST_INIT {NULL, NULL} +#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0 }void raw_object_store_clear (struct raw_object_store *o); +extern struct packed_git { +struct packed_git *next; +struct list_head mru; +struct pack_window *windows; +off_t pack_size; +const void *index_data; +size_t index_size; +uint32_t num_objects; +uint32_t num_bad_objects; +unsigned char *bad_object_sha1; +int index_version; +time_t mtime; +int pack_fd; +unsigned pack_local:1 , + pack_keep:1 , + freshened:1 , + do_not_close:1 , + pack_promisor:1 ; +unsigned char sha1[20 ]; +struct revindex_entry *revindex; +/* something like ".git/objects/pack/xxxxx.pack" */ +char pack_name[FLEX_ARRAY ];/* more */ +} *packed_git; +#endif /* OBJECT_STORE_H */
object.c +7 -0
@@ -466,4 +466,11 @@ void raw_object_store_clear(struct raw_object_store *o)free_alt_odbs (o); o->alt_odb_tail =NULL ; + +while (!list_empty (&o->packed_git_mru)) +list_del (&o->packed_git_mru); +/* +* TODO: call close_all_packs once migrated to +* take an object store argument +*/ }
pack-bitmap.c +3 -1
@@ -5,11 +5,13 @@#include "revision.h" #include "progress.h" #include "list-objects.h" +#include "object-store.h" #include "pack.h" #include "pack-bitmap.h" #include "pack-revindex.h" #include "pack-objects.h" #include "packfile.h" +#include "repository.h" /* * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +337,7 @@ static int open_pack_bitmap(void)assert (!bitmap_git.map && !bitmap_git.loaded);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (open_pack_bitmap_1 (p) ==0 ) ret =0 ; }
pack-check.c +1 -0
@@ -1,5 +1,6 @@#include "cache.h" #include "pack.h" +#include "object-store.h" #include "pack-revindex.h" #include "progress.h" #include "packfile.h"
pack-revindex.c +1 -0
@@ -1,5 +1,6 @@#include "cache.h" #include "pack-revindex.h" +#include "object-store.h" /* * Pack index for existing packs give us easy access to the offsets into
packfile.c +20 -19
@@ -7,6 +7,7 @@#include "packfile.h" #include "delta.h" #include "list.h" +#include "object-store.h" #include "streaming.h" #include "sha1-lookup.h" #include "commit.h"
@@ -45,8 +46,6 @@ static unsigned int pack_open_fds;static unsigned int pack_max_fds;static size_t peak_pack_mapped;static size_t pack_mapped; -struct packed_git *packed_git; -LIST_HEAD (packed_git_mru);#define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; }
@@ -246,7 +245,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -316,7 +315,7 @@ void close_all_packs(void) {struct packed_git *p; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)if (p->do_not_close)die ("BUG: want to close pack marked 'do-not-close'" );else
@@ -384,7 +383,7 @@ static int close_one_pack(void)struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -686,8 +685,8 @@ void install_packed_git(struct packed_git *pack)if (pack->pack_fd !=-1 ) pack_open_fds++; - pack->next = packed_git; - packed_git = pack; + pack->next = the_repository->objects.packed_git; + the_repository->objects.packed_git = pack; }void (*report_garbage)(unsigned seen_bits,const char *path);
@@ -769,7 +768,8 @@ static void prepare_packed_git_one(char *objdir, int local) base_len = path.len;if (strip_suffix_mem (path.buf, &base_len,".idx" )) {/* Don't reopen a pack we already have. */ -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {size_t len; if (strip_suffix(p->pack_name, ".pack", &len) && len == base_len &&
@@ -820,7 +820,7 @@ unsigned long approximate_object_count(void)prepare_packed_git (); count =0 ; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (open_pack_index (p))continue ; count += p->num_objects;
@@ -869,18 +869,19 @@ static int sort_pack(const void *a_, const void *b_)static void rearrange_packed_git (void ) { - packed_git =llist_mergesort (packed_git, get_next_packed_git, - set_next_packed_git, sort_pack); + the_repository->objects.packed_git =llist_mergesort ( + the_repository->objects.packed_git, get_next_packed_git, + set_next_packed_git, sort_pack); }static void prepare_packed_git_mru (void ) {struct packed_git *p; -INIT_LIST_HEAD (&packed_git_mru); +INIT_LIST_HEAD (&the_repository->objects.packed_git_mru); -for (p = packed_git; p; p = p->next) -list_add_tail (&p->mru, &packed_git_mru); +for (p = the_repository->objects.packed_git; p; p = p->next) +list_add_tail (&p->mru, &the_repository->objects.packed_git_mru); }static int prepare_packed_git_run_once =0 ;
@@ -1014,7 +1015,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)struct packed_git *p;unsigned i; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)for (i =0 ; i < p->num_bad_objects; i++)if (!hashcmp (sha1, p->bad_object_sha1 +20 * i))return p;
@@ -1845,13 +1846,13 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)struct list_head *pos;prepare_packed_git (); -if (!packed_git) +if (!the_repository->objects.packed_git)return 0 ; -list_for_each (pos, &packed_git_mru) { +list_for_each (pos, &the_repository->objects.packed_git_mru) { struct packed_git *p = list_entry(pos, struct packed_git, mru);if (fill_pack_entry (sha1, e, p)) { -list_move (&p->mru, &packed_git_mru); +list_move (&p->mru, &the_repository->objects.packed_git_mru);return 1 ; } }
@@ -1898,7 +1899,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)int pack_errors =0 ;prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ; if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
reachable.c +1 -0
@@ -1,5 +1,6 @@#include "cache.h" #include "refs.h" +#include "object-store.h" #include "tag.h" #include "commit.h" #include "blob.h"
repository.c +3 -0
@@ -144,6 +144,9 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree) repo->ignore_env =1 ; + repo->objects.ignore_env =1 ; +INIT_LIST_HEAD (&repo->objects.packed_git_mru); +if (repo_init_gitdir (repo, gitdir)) goto error;
server-info.c +4 -2
@@ -1,5 +1,7 @@#include "cache.h" +#include "repository.h" #include "refs.h" +#include "object-store.h" #include "object.h" #include "commit.h" #include "tag.h"
@@ -200,7 +202,7 @@ static void init_pack_info(const char *infofile, int force) objdirlen =strlen (objdir);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {/* we ignore things on alternate path since they are * not available to the pullers in general. */
@@ -210,7 +212,7 @@ static void init_pack_info(const char *infofile, int force) } num_pack = i; info =xcalloc (num_pack,sizeof (struct pack_info *)); -for (i =0 , p = packed_git; p; p = p->next) { +for (i =0 , p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ; info[i] =xcalloc (1 ,sizeof (struct pack_info ));
sha1_name.c +3 -2
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)struct packed_git *p;prepare_packed_git (); -for (p = packed_git; p && !ds->ambiguous; p = p->next) +for (p = the_repository->objects.packed_git; p && !ds->ambiguous; + p = p->next)unique_in_pack (p, ds); }
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)struct packed_git *p;prepare_packed_git (); -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)find_abbrev_len_for_pack (p, mad); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 04/16] object-store: move packed_git and packed_git_mru to object store
Quoted 3 lines · Stefan Beller
@@ @@
-packed_git_mru
+the_repository->objects.packed_git_mru
Regarding this...
Quoted 34 lines
object-store.h +11
-1@@ -2,6 +2,7 @@#define OBJECT_STORE_H #include "cache.h" +#include "list.h" struct raw_object_store { +struct packed_git *packed_git; +/* +* A most-recently-used ordered version of the packed_git list, which can +* be iterated instead of packed_git (and marked via mru_mark). +*/ +struct list_head packed_git_mru; +struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail;unsigned ignore_env :1 ; }; -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL, 0 } + +#define MRU_LIST_INIT {NULL, NULL} +#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0 }packfile.c +0
-0
packfile.c line 8
Quoted 2 lines
packfile.c +0
-2@@ -7,6 +7,7 @@ -... -LIST_HEAD (packed_git_mru);
Given that the definition of LIST_HEAD() is
/* Define a variable with the head and tail of the list. */
#define LIST_HEAD(name) \
struct list_head name = { &(name), &(name) }doesn't the updated definition of RAW_OBJECT_STORE_INIT look fishy?
PATCH v2 5/16 object-store: close all packs upon clearing the object store
In reply to Stefan Beller--- builtin/am.c | 2 +- builtin/clone.c | 2 +- builtin/fetch.c | 2 +- builtin/merge.c | 2 +- builtin/receive-pack.c | 2 +- object.c | 6 ++---- packfile.c | 4 ++-- packfile.h | 2 +- 8 files changed, 10 insertions(+), 12 deletions(-)
builtin/am.c +1 -1
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)*/ if (!state->rebasing) {am_destroy (state); -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }
builtin/clone.c +1 -1
@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)transport_disconnect (transport);if (option_dissociate) { -close_all_packs (); +close_all_packs (&the_repository->objects);dissociate_from_references (); }
builtin/fetch.c +1 -1
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)string_list_clear (&list,0 ); -close_all_packs (); +close_all_packs (&the_repository->objects);argv_array_pushl (&argv_gc_auto,"gc" ,"--auto" ,NULL );if (verbosity <0 )
builtin/merge.c +1 -1
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,* We ignore errors in 'gc --auto', since the * user should see them. */ -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }
builtin/receive-pack.c +1 -1
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix) proc.git_cmd =1 ; proc.argv = argv_gc_auto; -close_all_packs (); +close_all_packs (&the_repository->objects);if (!start_command (&proc)) {if (use_sideband)copy_to_sideband (proc.err,-1 ,NULL );
object.c +2 -4
@@ -4,6 +4,7 @@#include "tree.h" #include "commit.h" #include "tag.h" +#include "packfile.h" static struct object **obj_hash;static int nr_objs, obj_hash_size;
@@ -469,8 +470,5 @@ void raw_object_store_clear(struct raw_object_store *o)while (!list_empty (&o->packed_git_mru))list_del (&o->packed_git_mru); -/* -* TODO: call close_all_packs once migrated to -* take an object store argument -*/ +close_all_packs (o); }
packfile.c +2 -2
@@ -311,11 +311,11 @@ static void close_pack(struct packed_git *p)close_pack_index (p); } -void close_all_packs (void ) +void close_all_packs (struct raw_object_store *o) {struct packed_git *p; -for (p = the_repository->objects.packed_git; p; p = p->next) +for (p = o->packed_git; p; p = p->next)if (p->do_not_close)die ("BUG: want to close pack marked 'do-not-close'" );else
packfile.h +1 -1
@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void close_pack_windows (struct packed_git *); -extern void close_all_packs (void ); +extern void close_all_packs (struct raw_object_store *o);extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 6/16 pack: move prepare_packed_git_run_once to object store
In reply to Stefan BellerEach repository's object store can be initialized independently, so they must not share a run_once variable.
--- object-store.h | 8 +++++++- packfile.c | 7 +++---- 2 files changed, 10 insertions(+), 5 deletions(-)
object-store.h +7 -1
@@ -22,10 +22,16 @@ struct raw_object_store {struct alternate_object_database **alt_odb_tail; unsigned ignore_env : 1; + +/* +* Whether packed_git has already been populated with this repository's +* packs. +*/ + unsigned packed_git_initialized : 1; };#define MRU_LIST_INIT {NULL, NULL} -#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0 } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +3 -4
@@ -884,12 +884,11 @@ static void prepare_packed_git_mru(void)list_add_tail (&p->mru, &the_repository->objects.packed_git_mru); } -static int prepare_packed_git_run_once =0 ;void prepare_packed_git (void ) {struct alternate_object_database *alt; -if (prepare_packed_git_run_once) +if (the_repository->objects.packed_git_initialized)return ;prepare_packed_git_one (get_object_directory (),1 );prepare_alt_odb ();
@@ -897,13 +896,13 @@ void prepare_packed_git(void)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();prepare_packed_git_mru (); - prepare_packed_git_run_once =1 ; + the_repository->objects.packed_git_initialized =1 ; }void reprepare_packed_git (void ) { approximate_object_count_valid =0 ; - prepare_packed_git_run_once =0 ; + the_repository->objects.packed_git_initialized =0 ;prepare_packed_git (); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 7/16 pack: move approximate object count to object store
In reply to Stefan BellerThe approximate_object_count() function maintains a rough count of objects in a repository to estimate how long object name abbreviates should be. Object names are scoped to a repository and the appropriate length may differ by repository, so the object count should not be global.
--- object-store.h | 10 +++++++++- packfile.c | 11 +++++------ 2 files changed, 14 insertions(+), 7 deletions(-)
object-store.h +9 -1
@@ -23,6 +23,14 @@ struct raw_object_store { unsigned ignore_env : 1; +/* +* A fast, rough count of the number of objects in the repository. +* These two fields are not meant for direct access. Use +* approximate_object_count() instead. +*/ +unsigned long approximate_object_count; + unsigned approximate_object_count_valid : 1; +/* * Whether packed_git has already been populated with this repository's * packs.
@@ -31,7 +39,7 @@ struct raw_object_store { };#define MRU_LIST_INIT {NULL, NULL} -#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0, 0 } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0, 0, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +5 -6
@@ -803,8 +803,6 @@ static void prepare_packed_git_one(char *objdir, int local)strbuf_release (&path); } -static int approximate_object_count_valid; -/* * Give a fast, rough count of the number of objects in the repository. This * ignores loose objects completely. If you have a lot of them, then either
@@ -814,8 +812,8 @@ static int approximate_object_count_valid;*/ unsigned long approximate_object_count (void ) { -static unsigned long count; -if (!approximate_object_count_valid) { +if (!the_repository->objects.approximate_object_count_valid) { +unsigned long count;struct packed_git *p;prepare_packed_git ();
@@ -825,8 +823,9 @@ unsigned long approximate_object_count(void)continue ; count += p->num_objects; } + the_repository->objects.approximate_object_count = count; } -return count; +return the_repository->objects.approximate_object_count; }static void *get_next_packed_git (const void *p)
@@ -901,7 +900,7 @@ void prepare_packed_git(void)void reprepare_packed_git (void ) { - approximate_object_count_valid =0 ; + the_repository->objects.approximate_object_count_valid =0 ; the_repository->objects.packed_git_initialized =0 ;prepare_packed_git (); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 8/16 sha1_file: add raw_object_store argument to alt_odb_usable
In reply to Stefan BellerAdd a raw_object_store to alt_odb_usable to be more specific about which repository to act on. The choice of the repository is delegated to its only caller link_alt_odb_entry.
--- sha1_file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
sha1_file.c +5 -3
@@ -346,7 +346,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,/* * Return non-zero iff the path is usable as an alternate object database. */ -static int alt_odb_usable (struct strbuf *path,const char *normalized_objdir) +static int alt_odb_usable (struct raw_object_store *o, +struct strbuf *path, +const char *normalized_objdir) {struct alternate_object_database *alt;
@@ -362,7 +364,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)* Prevent the common mistake of listing the same * thing twice, or object directory itself. */ -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +for (alt = o->alt_odb_list; alt; alt = alt->next) {if (!fspathcmp (path->buf, alt->path))return 0 ; }
@@ -414,7 +416,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,while (pathbuf.len && pathbuf.buf[pathbuf.len -1 ] =='/' )strbuf_setlen (&pathbuf, pathbuf.len -1 ); -if (!alt_odb_usable (&pathbuf, normalized_objdir)) { +if (!alt_odb_usable (&the_repository->objects, &pathbuf, normalized_objdir)) {strbuf_release (&pathbuf);return -1 ; }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 9/16 sha1_file: allow link_alt_odb_entries to handle arbitrary object stores
In reply to Stefan BellerActually this also allows read_info_alternates and link_alt_odb_entry to handle arbitrary object stores, but link_alt_odb_entries is the most interesting function in this set of functions, hence the commit subject.
These functions span a strongly connected component in the function graph, i.e. the recursive call chain might look like
-> link_alt_odb_entries
-> link_alt_odb_entry
-> read_info_alternates
-> link_alt_odb_entriesThat is why we need to convert all these functions at the same time.
--- cache.h | 4 ++++ sha1_file.c | 42 +++++++++++++++++++++++++----------------- 2 files changed, 29 insertions(+), 17 deletions(-)
cache.h +4 -0
@@ -1593,6 +1593,10 @@ struct alternate_object_database {char loose_objects_subdir_seen[256 ];struct oid_array loose_objects_cache; +/* +* Path to the alternative object store. If this is a relative path, +* it is relative to the current working directory. +*/ char path[FLEX_ARRAY ]; };extern void prepare_alt_odb (void );
sha1_file.c +25 -17
111 lines not on this page · raw message
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 10/16 sha1_file: allow prepare_alt_odb to handle arbitrary object stores
In reply to Stefan Beller--- builtin/fsck.c | 2 +- cache.h | 1 - object-store.h | 2 ++ packfile.c | 2 +- sha1_file.c | 23 +++++++++++------------ sha1_name.c | 3 ++- 6 files changed, 17 insertions(+), 16 deletions(-)
builtin/fsck.c +1 -1
@@ -717,7 +717,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) }else {fsck_object_dir (get_object_directory ()); -prepare_alt_odb (); +prepare_alt_odb (&the_repository->objects);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)fsck_object_dir (alt->path);
cache.h +0 -1
@@ -1599,7 +1599,6 @@ struct alternate_object_database {*/ char path[FLEX_ARRAY ]; }; -extern void prepare_alt_odb (void );extern char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);extern int foreach_alt_odb (alt_odb_fn ,void *);
object-store.h +2 -0
@@ -67,4 +67,6 @@ extern struct packed_git { char pack_name[FLEX_ARRAY]; /* more */ } *packed_git; +void prepare_alt_odb (struct raw_object_store *o); +#endif /* OBJECT_STORE_H */
packfile.c +1 -1
@@ -890,7 +890,7 @@ void prepare_packed_git(void)if (the_repository->objects.packed_git_initialized)return ;prepare_packed_git_one (get_object_directory (),1 ); -prepare_alt_odb (); +prepare_alt_odb (&the_repository->objects);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();
sha1_file.c +11 -12
@@ -23,6 +23,7 @@#include "sha1-lookup.h" #include "bulk-checkin.h" #include "repository.h" +#include "object-store.h" #include "streaming.h" #include "dir.h" #include "list.h"
@@ -577,7 +578,7 @@ void add_to_alternates_memory(const char *reference)* Make sure alternates are initialized, or else our entry may be * overwritten when they are. */ -prepare_alt_odb (); +prepare_alt_odb (&the_repository->objects);link_alt_odb_entries (&the_repository->objects, reference,'\n' ,NULL ,0 );
@@ -663,7 +664,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)struct alternate_object_database *ent;int r =0 ; -prepare_alt_odb (); +prepare_alt_odb (&the_repository->objects);for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) { r =fn (ent, cb);if (r)
@@ -672,21 +673,19 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb (void ) +void prepare_alt_odb (struct raw_object_store *o) {const char *alt; -if (the_repository->objects.alt_odb_tail) +if (o->alt_odb_tail)return ; alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - the_repository->objects.alt_odb_tail = - &the_repository->objects.alt_odb_list; -link_alt_odb_entries (&the_repository->objects, alt, -PATH_SEP ,NULL ,0 ); + o->alt_odb_tail = &o->alt_odb_list; +link_alt_odb_entries (o, alt,PATH_SEP ,NULL ,0 ); -read_info_alternates (&the_repository->objects,get_object_directory (),0 ); +read_info_alternates (o, o->objectdir,0 ); }/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
@@ -726,7 +725,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)static int check_and_freshen_nonlocal (const unsigned char *sha1,int freshen) {struct alternate_object_database *alt; -prepare_alt_odb (); +prepare_alt_odb (&the_repository->objects);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {const char *path =alt_sha1_path (alt, sha1);if (check_and_freshen_file (path, freshen))
@@ -885,7 +884,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,if (!lstat (*path, st))return 0 ; -prepare_alt_odb (); +prepare_alt_odb (&the_repository->objects); errno =ENOENT ;for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);
@@ -916,7 +915,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)return fd; most_interesting_errno = errno; -prepare_alt_odb (); +prepare_alt_odb (&the_repository->objects);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);
sha1_name.c +2 -1
@@ -11,6 +11,7 @@#include "sha1-array.h" #include "packfile.h" #include "repository.h" +#include "object-store.h" static int get_oid_oneline (const char *,struct object_id *,struct commit_list *);
@@ -353,7 +354,7 @@ static int init_object_disambiguation(const char *name, int len, ds->len = len; ds->hex_pfx[len] ='\0' ; -prepare_alt_odb (); +prepare_alt_odb (&the_repository->objects);return 0 ; }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 11/16 sha1_file: allow sha1_file_name to handle arbitrary object stores
In reply to Stefan BellerWhile at it, move the declaration to object-store.h, where it should be easier to find.
--- cache.h | 6 ------ http-walker.c | 3 ++- http.c | 5 ++--- object-store.h | 6 ++++++ sha1_file.c | 12 ++++++------ 5 files changed, 16 insertions(+), 16 deletions(-)
cache.h +0 -6
@@ -961,12 +961,6 @@ extern void check_repository_format(void);#define DATA_CHANGED 0x0020#define TYPE_CHANGED 0x0040 -/* -* Put in `buf` the name of the file in the local object database that -* would be used to store a loose object with the specified sha1. -*/ -extern void sha1_file_name (struct strbuf *buf,const unsigned char *sha1); -/* * Return an abbreviated sha1 unique within this repository's object database. * The result will be at least `len` characters long, and will be NUL
http-walker.c +2 -1
@@ -1,4 +1,5 @@#include "cache.h" +#include "repository.h" #include "commit.h" #include "walker.h" #include "http.h"
@@ -546,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1) ret =error ("File %s has bad hash" , hex); }else if (req->rename <0 ) {struct strbuf buf =STRBUF_INIT ; -sha1_file_name (&buf, req->sha1); +sha1_file_name (&the_repository->objects, &buf, req->sha1); ret =error ("unable to write sha1 filename %s" , buf.buf);strbuf_release (&buf); }
http.c +2 -3
@@ -2247,7 +2247,7 @@ struct http_object_request *new_http_object_request(const char *base_url,hashcpy (freq->sha1, sha1); freq->localfile =-1 ; -sha1_file_name (&filename, sha1); +sha1_file_name (&the_repository->objects, &filename, sha1);snprintf (freq->tmpfile,sizeof (freq->tmpfile),"%s.temp" , filename.buf);
@@ -2396,8 +2396,7 @@ int finish_http_object_request(struct http_object_request *freq)unlink_or_warn (freq->tmpfile);return -1 ; } - -sha1_file_name (&filename, freq->sha1); +sha1_file_name (&the_repository->objects, &filename, freq->sha1); freq->rename =finalize_object_file (freq->tmpfile, filename.buf);strbuf_release (&filename);
object-store.h +6 -0
@@ -67,6 +67,12 @@ extern struct packed_git { char pack_name[FLEX_ARRAY]; /* more */ } *packed_git; +/* +* Put in `buf` the name of the file in the local object database that +* would be used to store a loose object with the specified sha1. +*/ +void sha1_file_name (struct raw_object_store *o,struct strbuf *buf,const unsigned char *sha1); +void prepare_alt_odb (struct raw_object_store *o);#endif /* OBJECT_STORE_H */
sha1_file.c +6 -6
@@ -323,9 +323,9 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1) } } -void sha1_file_name (struct strbuf *buf,const unsigned char *sha1) +void sha1_file_name (struct raw_object_store *o,struct strbuf *buf,const unsigned char *sha1) { -strbuf_addstr (buf,get_object_directory ()); +strbuf_addstr (buf, o->objectdir);strbuf_addch (buf,'/' );fill_sha1_path (buf, sha1); }
@@ -717,7 +717,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&buf, sha1); +sha1_file_name (&the_repository->objects, &buf, sha1);return check_and_freshen_file (buf.buf, freshen); }
@@ -878,7 +878,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&buf, sha1); +sha1_file_name (&the_repository->objects, &buf, sha1); *path = buf.buf;if (!lstat (*path, st))
@@ -907,7 +907,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&buf, sha1); +sha1_file_name (&the_repository->objects, &buf, sha1); *path = buf.buf; fd =git_open (*path);
@@ -1592,7 +1592,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,static struct strbuf filename =STRBUF_INIT ;strbuf_reset (&filename); -sha1_file_name (&filename, sha1); +sha1_file_name (&the_repository->objects, &filename, sha1); fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 12/16 sha1_file: allow stat_sha1_file to handle arbitrary object stores
In reply to Stefan Beller--- sha1_file.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
sha1_file.c +8 -7
@@ -871,22 +871,23 @@ int git_open_cloexec(const char *name, int flags)* Note that it may point to static storage and is only valid until another * call to sha1_file_name(), etc. */ -static int stat_sha1_file (const unsigned char *sha1,struct stat *st, -const char **path) +static int stat_sha1_file (struct raw_object_store *o,const unsigned char *sha1, +struct stat *st,const char **path) {struct alternate_object_database *alt;static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&the_repository->objects, &buf, sha1); +sha1_file_name (o, &buf, sha1); *path = buf.buf;if (!lstat (*path, st))return 0 ; -prepare_alt_odb (&the_repository->objects); +prepare_alt_odb (o); + errno =ENOENT ; -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +for (alt = o->alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);if (!lstat (*path, st))return 0 ;
@@ -1178,7 +1179,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {const char *path;struct stat st; -if (stat_sha1_file (sha1, &st, &path) <0 ) +if (stat_sha1_file (&the_repository->objects, sha1, &st, &path) <0 )return -1 ;if (oi->disk_sizep) *oi->disk_sizep = st.st_size;
@@ -1392,7 +1393,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,die ("replacement %s not found for %s" ,sha1_to_hex (repl),sha1_to_hex (sha1)); -if (!stat_sha1_file (repl, &st, &path)) +if (!stat_sha1_file (&the_repository->objects, repl, &st, &path))die ("loose object %s (stored in %s) is corrupt" ,sha1_to_hex (repl), path);
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 13/16 sha1_file: allow open_sha1_file to handle arbitrary object stores
In reply to Stefan Beller--- sha1_file.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
sha1_file.c +6 -5
@@ -900,7 +900,8 @@ static int stat_sha1_file(struct raw_object_store *o, const unsigned char *sha1,* Like stat_sha1_file(), but actually open the object and return the * descriptor. See the caveats on the "path" parameter above. */ -static int open_sha1_file (const unsigned char *sha1,const char **path) +static int open_sha1_file (struct raw_object_store *o, +const unsigned char *sha1,const char **path) {int fd;struct alternate_object_database *alt;
@@ -908,7 +909,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&the_repository->objects, &buf, sha1); +sha1_file_name (o, &buf, sha1); *path = buf.buf; fd =git_open (*path);
@@ -916,8 +917,8 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)return fd; most_interesting_errno = errno; -prepare_alt_odb (&the_repository->objects); -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +prepare_alt_odb (o); +for (alt = o->alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);if (fd >=0 )
@@ -943,7 +944,7 @@ static void *map_sha1_file_1(const char *path,if (path) fd =git_open (path);else - fd =open_sha1_file (sha1, &path); + fd =open_sha1_file (&the_repository->objects, sha1, &path); map =NULL ;if (fd >=0 ) {struct stat st;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 14/16 sha1_file: allow map_sha1_file_1 to handle arbitrary object stores
In reply to Stefan Beller--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -934,9 +934,8 @@ static int open_sha1_file(struct raw_object_store *o,* Map the loose object at "path" if it is not NULL, or the path found by * searching for a loose object named "sha1". */ -static void *map_sha1_file_1 (const char *path, -const unsigned char *sha1, -unsigned long *size) +static void *map_sha1_file_1 (struct raw_object_store *o,const char *path, +const unsigned char *sha1,unsigned long *size) {void *map;int fd;
@@ -944,7 +943,7 @@ static void *map_sha1_file_1(const char *path,if (path) fd =git_open (path);else - fd =open_sha1_file (&the_repository->objects, sha1, &path); + fd =open_sha1_file (o, sha1, &path); map =NULL ;if (fd >=0 ) {struct stat st;
@@ -965,7 +964,7 @@ static void *map_sha1_file_1(const char *path,void *map_sha1_file (const unsigned char *sha1,unsigned long *size) { -return map_sha1_file_1 (NULL , sha1, size); +return map_sha1_file_1 (&the_repository->objects,NULL , sha1, size); } static int unpack_sha1_short_header(git_zstream *stream,
@@ -2195,7 +2194,7 @@ int read_loose_object(const char *path, *contents =NULL ; - map =map_sha1_file_1 (path,NULL , &mapsize); + map =map_sha1_file_1 (&the_repository->objects, path,NULL , &mapsize);if (!map) {error_errno ("unable to mmap %s" , path); goto out;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 15/16 sha1_file: allow map_sha1_file to handle arbitrary object stores
In reply to Stefan Beller--- cache.h | 1 - object-store.h | 2 ++ sha1_file.c | 7 ++++--- streaming.c | 5 ++++- 4 files changed, 10 insertions(+), 5 deletions(-)
cache.h +0 -1
@@ -1242,7 +1242,6 @@ extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned cextern int force_object_loose (const unsigned char *sha1,time_t mtime);extern int git_open_cloexec (const char *name,int flags);#define git_open (name) git_open_cloexec(name, O_RDONLY) -extern void *map_sha1_file (const unsigned char *sha1,unsigned long *size);extern int unpack_sha1_header (git_zstream *stream,unsigned char *map,unsigned long mapsize,void *buffer,unsigned long bufsiz);extern int parse_sha1_header (const char *hdr,unsigned long *sizep);
object-store.h +2 -0
@@ -73,6 +73,8 @@ extern struct packed_git {*/ void sha1_file_name (struct raw_object_store *o,struct strbuf *buf,const unsigned char *sha1); +void *map_sha1_file (struct raw_object_store *o,const unsigned char *sha1,unsigned long *size); +void prepare_alt_odb (struct raw_object_store *o);#endif /* OBJECT_STORE_H */
sha1_file.c +4 -3
@@ -962,9 +962,10 @@ static void *map_sha1_file_1(struct raw_object_store *o, const char *path,return map; } -void *map_sha1_file (const unsigned char *sha1,unsigned long *size) +void *map_sha1_file (struct raw_object_store *o, +const unsigned char *sha1,unsigned long *size) { -return map_sha1_file_1 (&the_repository->objects,NULL , sha1, size); +return map_sha1_file_1 (o,NULL , sha1, size); } static int unpack_sha1_short_header(git_zstream *stream,
@@ -1186,7 +1187,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,return 0 ; } - map =map_sha1_file (sha1, &mapsize); + map =map_sha1_file (&the_repository->objects, sha1, &mapsize);if (!map)return -1 ;
streaming.c +4 -1
@@ -3,6 +3,8 @@*/ #include "cache.h" #include "streaming.h" +#include "repository.h" +#include "object-store.h" #include "packfile.h" enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {static open_method_decl (loose) { - st->u.loose.mapped =map_sha1_file (sha1, &st->u.loose.mapsize); + st->u.loose.mapped =map_sha1_file (&the_repository->objects, + sha1, &st->u.loose.mapsize);if (!st->u.loose.mapped)return -1 ; if ((unpack_sha1_header(&st->z,
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v2 16/16 sha1_file: allow sha1_loose_object_info to handle arbitrary object stores
In reply to Stefan Beller--- sha1_file.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
sha1_file.c +5 -4
@@ -1154,7 +1154,8 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -static int sha1_loose_object_info (const unsigned char *sha1, +static int sha1_loose_object_info (struct raw_object_store *o, +const unsigned char *sha1,struct object_info *oi,int flags) {
@@ -1180,14 +1181,14 @@ static int sha1_loose_object_info(const unsigned char *sha1,if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {const char *path;struct stat st; -if (stat_sha1_file (&the_repository->objects, sha1, &st, &path) <0 ) +if (stat_sha1_file (o, sha1, &st, &path) <0 )return -1 ;if (oi->disk_sizep) *oi->disk_sizep = st.st_size;return 0 ; } - map =map_sha1_file (&the_repository->objects, sha1, &mapsize); + map =map_sha1_file (o, sha1, &mapsize);if (!map)return -1 ;
@@ -1275,7 +1276,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,break ;/* Most likely it's a loose object. */ -if (!sha1_loose_object_info (real, oi, flags)) +if (!sha1_loose_object_info (&the_repository->objects, real, oi, flags))return 0 ;/* Not a loose object; someone else may have just packed it. */
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Hi,
Quoted 9 lines · Stefan Beller
v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global, which means there is just one patch per function that is converted. As this follows the same structure of the previous series, I am still confident there is no hidden dependencies to globals outside the object store in these converted functions.
* rebased on top of current master, resolving the merge conflicts.
I think I used thelist.hAPIs right, but please double check.
For what it's worth, I think I prefer v1. I put some comments on why on patch 0 of v1 and would be interested in your thoughts on them (e.g. as a reply to that). I also think that even if we want to switch to a style that passes around object_store separately from repository, it is easier to do the migration in two steps: first get rid of hidden dependencies on the_repository, then do the (simpler) automatic migration from
f(the_repository)
to
f(the_repository->object_store)
*afterwards*.
Thoughts?
Thanks,
Jonathan
Quoted 24 lines · Jonathan Nieder
Hi,
Quoted 9 lines · Stefan Beller
v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global, which means there is just one patch per function that is converted. As this follows the same structure of the previous series, I am still confident there is no hidden dependencies to globals outside the object store in these converted functions.
* rebased on top of current master, resolving the merge conflicts.
I think I used thelist.hAPIs right, but please double check.For what it's worth, I think I prefer v1. I put some comments on why on patch 0 of v1 and would be interested in your thoughts on them (e.g. as a reply to that). I also think that even if we want to switch to a style that passes around
object_storeseparately from repository, it is easier to do the migration in two steps: first get rid of hidden dependencies onthe_repository, then do the (simpler) automatic migration from
f(the_repository)
Quoted 7 lines
to
f(the_repository->object_store)*afterwards*.
Thoughts?
I would prefer to not spend more time on these conversions. If Duy and you would come to a conclusion to either pick this or the previous version I would be happy.
I do not see the benefit in splitting up the series even further and do this multistep f(repo) -> f(object store), as the cost in potential merge conflicts is too high. Note that brian just sent another object id conversion series, also touching sha1_file.c, which I am sure will produce merge conflicts for Junio.
For the next part 2 and onwards I'd be happy to take either this strategy or Duys strategy as requested.
Stefan
Quoted 37 lines · Stefan Beller
Quoted 24 lines · Jonathan Nieder
Hi,
Quoted 9 lines · Stefan Beller
v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global, which means there is just one patch per function that is converted. As this follows the same structure of the previous series, I am still confident there is no hidden dependencies to globals outside the object store in these converted functions.
* rebased on top of current master, resolving the merge conflicts.
I think I used thelist.hAPIs right, but please double check.For what it's worth, I think I prefer v1. I put some comments on why on patch 0 of v1 and would be interested in your thoughts on them (e.g. as a reply to that). I also think that even if we want to switch to a style that passes around
object_storeseparately from repository, it is easier to do the migration in two steps: first get rid of hidden dependencies onthe_repository, then do the (simpler) automatic migration from
f(the_repository)Quoted 7 lines
to
f(the_repository->object_store)*afterwards*.
Thoughts?
I would prefer to not spend more time on these conversions. If Duy and you would come to a conclusion to either pick this or the previous version I would be happy.
Quoted 8 lines
I do not see the benefit in splitting up the series even further and do this multistep f(repo) -> f(object store), as the cost in potential merge conflicts is too high. Note that brian just sent another object id conversion series, also touching
sha1_file.c, which I am sure will produce merge conflicts for Junio.For the next part 2 and onwards I'd be happy to take either this strategy or Duys strategy as requested.
I think Jonathan is trying to point out that converting to f(repo) maybe easier than converting to f(repo->object_store) upfront which would make it easier to write the patches (which most of them are already written) and to review because you can use the #define trick to make some sort of guarantees.
After we have successfully completed the migration to f(repo), then we can revisit the subsystems which want to have a clearer abstraction layer and make the jump to f(repo->object_store).
Signature · 1 line
Brandon Williams
Quoted 44 lines · Brandon Williams
Quoted 43 lines · Stefan Beller
Quoted 24 lines · Jonathan Nieder
Hi,
Quoted 9 lines · Stefan Beller
v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global, which means there is just one patch per function that is converted. As this follows the same structure of the previous series, I am still confident there is no hidden dependencies to globals outside the object store in these converted functions.
* rebased on top of current master, resolving the merge conflicts.
I think I used thelist.hAPIs right, but please double check.For what it's worth, I think I prefer v1. I put some comments on why on patch 0 of v1 and would be interested in your thoughts on them (e.g. as a reply to that). I also think that even if we want to switch to a style that passes around
object_storeseparately from repository, it is easier to do the migration in two steps: first get rid of hidden dependencies onthe_repository, then do the (simpler) automatic migration from
f(the_repository)Quoted 7 lines
to
f(the_repository->object_store)*afterwards*.
Thoughts?
I would prefer to not spend more time on these conversions. If Duy and you would come to a conclusion to either pick this or the previous version I would be happy.
I do not see the benefit in splitting up the series even further and do this multistep f(repo) -> f(object store), as the cost in potential merge conflicts is too high. Note that brian just sent another object id conversion series, also touching
sha1_file.c, which I am sure will produce merge conflicts for Junio.
Quoted 5 lines
Quoted 2 lines
For the next part 2 and onwards I'd be happy to take either this strategy or Duys strategy as requested.
I think Jonathan is trying to point out that converting to f(repo) maybe easier than converting to
f(repo->object_store)upfront
I agree.
Quoted 2 lines
which would make
it easier to write the patches (which most of them are already written)
true, but for this series we also have the conversion to f(object_store) written already.
Quoted 2 lines
and to review because you can use the #define trick to make some sort of guarantees.
That is true, so it would be a tradeoff between reviewers and maintainers time?
Quoted 3 lines
After we have successfully completed the migration to f(repo), then we can revisit the subsystems which want to have a clearer abstraction layer and make the jump to
f(repo->object_store).
I would think we can take this series as-is and then move on with making f(repo) abstractions, eventually moving to f(specialized-subsystem) as those patches are not written yet (neither direct conversions, nor the repo trick; the patches I already have need adaption which takes enough time on its own.)
Quoted 3 lines
--
Brandon Williams
Quoted 10 lines · Jonathan Nieder
For what it's worth, I think I prefer v1. I put some comments on why on patch 0 of v1 and would be interested in your thoughts on them (e.g. as a reply to that). I also think that even if we want to switch to a style that passes around
object_storeseparately from repository, it is easier to do the migration in two steps: first get rid of hidden dependencies onthe_repository, then do the (simpler) automatic migration from
f(the_repository)
Quoted 7 lines
to
f(the_repository->object_store)*afterwards*.
Thoughts?
Are we envisioning the future in which one repository has more than one object-store (I am counting an object store and its alternates that are pointed by its $GIT_OBJECT_DIRECTORY/info/alternates as a single logical "object store")? If not, doing f(the_repository) migration, stopping there without f(the_repository->object_store) may perfectly be adequate, I would think.
Quoted 18 lines · Junio C Hamano
Quoted 16 lines · Jonathan Nieder
For what it's worth, I think I prefer v1. I put some comments on why on patch 0 of v1 and would be interested in your thoughts on them (e.g. as a reply to that). I also think that even if we want to switch to a style that passes around
object_storeseparately from repository, it is easier to do the migration in two steps: first get rid of hidden dependencies onthe_repository, then do the (simpler) automatic migration from
f(the_repository)to
f(the_repository->object_store)*afterwards*.
Quoted 9 lines
Quoted 1 line
Thoughts?
Are we envisioning the future in which one repository has more than one object-store (I am counting an object store and its alternates that are pointed by its $GIT_OBJECT_DIRECTORY/info/alternates as a single logical "object store")? If not, doing
f(the_repository)migration, stopping there withoutf(the_repository->object_store)may perfectly be adequate, I would think.
I do not envision that, maybe Christian Couder does? The partial clone world would be happy with just one object store per repo, I would think.
The step to take an object store would just add expressiveness to the code, which may help in understanding what part of the code is related to what other part of the code, so it may be a readability gain on its own?
Quoted 4 lines · Stefan Beller
The step to take an object store would just add expressiveness to the code, which may help in understanding what part of the code is related to what other part of the code, so it may be a readability gain on its own?
It certainly would allow you to have a standalone object store that is not associated with *any* repository, but if we are not using such a layout, I doubt it would be a readability gain to add excess and unexercised expressiveness to the code.
Quoted 11 lines · Junio C Hamano
Quoted 4 lines · Stefan Beller
The step to take an object store would just add expressiveness to the code, which may help in understanding what part of the code is related to what other part of the code, so it may be a readability gain on its own?
It certainly would allow you to have a standalone object store that is not associated with *any* repository, but if we are not using such a layout, I doubt it would be a readability gain to add excess and unexercised expressiveness to the code.
So you favor v1?
Duy seems to be ok with v1 too if there is consensus that it is best (or rather "if it makes Stefan's life hell, it's not worth doing.")[1].
I will try to resend that v1 shortly[2], as the only actual concern about the code was where one struct was defined[3]. All other discussion was meta-level, which direction to go.
Thanks for the clarification!
Stefan
[1] https://public-inbox.org/git/CACsJy8CPKESE8atc_eWdNVknQYp9T6ebwKwCdzLHyaFKH2BnZA@mail.gmail.com/
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/
v3 cover Moving global state into the repository object (part 1)
In reply to Stefan Bellerv3:
* reverted back to use the repository for most of the functions (including unduplicating 'ignore_env')
* rebased on master again (I lost that state when doing v2, as I did both rebase as well as conversion to object store in one go)
* one additional patch, that moves the alternates related things to object-store.h, such that inclusion of cache.h (in object-store.h) is not required any more.
v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global, which means there is just one patch per function that is converted. As this follows the same structure of the previous series, I am still confident there is no hidden dependencies to globals outside the object store in these converted functions.
* rebased on top of current master, resolving the merge conflicts.
I think I used the list.h APIs right, but please double check.
Thanks,
Stefan
v1:
This is a real take on the first part of the recent RFC[1].
Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories" might be a good breaking point for a first part at that RFC at patch 38. This series is smaller and contains only 26 patches as the patches in the big RFC were slightly out of order.
I developed this series partly by writing patches, but mostly by cherrypicking from that RFC on top of current master. I noticed no external conflicts apart from one addition to the repositories _INIT macro, which was easy to resolve.
Comments in the early range of that RFC were on 003 where Junio pointed out that the coccinelle patch ought to be not in contrib/coccinelle, so I put it in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.
brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself without any suggestion to include into this series[3].
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the ignore_env flag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.
Eric Sunshine brought up memory leaks with the RFC, and I would think to have plugged all holes.
[1] https://public-inbox.org/git/[email protected]/
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/
Thanks,
Stefan
Jonathan Nieder (2):
sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
sha1_file: allow sha1_loose_object_info to handle arbitrary
repositories
Stefan Beller (25):
repository: introduce raw object store field
object-store: migrate alternates struct and functions from cache.h
object-store: move alt_odb_list and alt_odb_tail to object store
object-store: free alt_odb_list
object-store: move packed_git and packed_git_mru to object store
object-store: close all packs upon clearing the object store
pack: move prepare_packed_git_run_once to object store
pack: move approximate object count to object store
sha1_file: add raw_object_store argument to alt_odb_usable
sha1_file: add repository argument to link_alt_odb_entry
sha1_file: add repository argument to read_info_alternates
sha1_file: add repository argument to link_alt_odb_entries
sha1_file: add repository argument to prepare_alt_odb
sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
sha1_file: allow prepare_alt_odb to handle arbitrary repositories
sha1_file: add repository argument to sha1_file_name
sha1_file: add repository argument to stat_sha1_file
sha1_file: add repository argument to open_sha1_file
sha1_file: add repository argument to map_sha1_file_1
sha1_file: add repository argument to map_sha1_file
sha1_file: add repository argument to sha1_loose_object_info
sha1_file: allow sha1_file_name to handle arbitrary repositories
sha1_file: allow stat_sha1_file to handle arbitrary repositories
sha1_file: allow open_sha1_file to handle arbitrary repositories
sha1_file: allow map_sha1_file to handle arbitrary repositories
builtin/am.c | 2 +- builtin/clone.c | 2 +- builtin/count-objects.c | 6 +- builtin/fetch.c | 2 +- builtin/fsck.c | 13 ++-- builtin/gc.c | 4 +- builtin/grep.c | 2 +- builtin/index-pack.c | 1 + builtin/merge.c | 2 +- builtin/pack-objects.c | 20 +++--- builtin/pack-redundant.c | 6 +- builtin/receive-pack.c | 3 +- cache.h | 87 ------------------------- environment.c | 5 +- fast-import.c | 6 +- http-backend.c | 6 +- http-push.c | 1 + http-walker.c | 4 +- http.c | 6 +- object-store.h | 134 +++++++++++++++++++++++++++++++++++++++ object.c | 27 ++++++++ pack-bitmap.c | 4 +- pack-check.c | 1 + pack-revindex.c | 1 + packfile.c | 64 +++++++++---------- packfile.h | 2 +- path.c | 2 +- reachable.c | 1 + repository.c | 19 ++++-- repository.h | 7 +- server-info.c | 6 +- sha1_file.c | 134 +++++++++++++++++++++------------------ sha1_name.c | 11 ++-- streaming.c | 5 +- 34 files changed, 365 insertions(+), 231 deletions(-) create mode 100644 object-store.h
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v3 1/27 repository: introduce raw object store field
The raw object store field will contain any objects needed for access to objects in a given repository.
This patch introduces the raw object store and populates it with the objectdir, which used to be part of the repository struct.
As the struct gains members, we'll also populate the function to clear the memory for these members.
In a later we'll introduce a struct object_parser, that will complement the object parsing in a repository struct: The raw object parser is the layer that will provide access to raw object content, while the higher level object parser code will parse raw objects and keeps track of parenthood and other object relationships using 'struct object'. For now only add the lower level to the repository struct.
--- builtin/grep.c | 2 +- environment.c | 5 +++-- object-store.h | 15 +++++++++++++++ object.c | 5 +++++ path.c | 2 +- repository.c | 17 +++++++++++++---- repository.h | 7 ++++--- 7 files changed, 42 insertions(+), 11 deletions(-) create mode 100644 object-store.h
builtin/grep.c +1 -1
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,* object. */ grep_read_lock (); -add_to_alternates_memory (submodule.objectdir); +add_to_alternates_memory (submodule.objects.objectdir);grep_read_unlock ();if (oid) {
environment.c +3 -2
@@ -13,6 +13,7 @@#include "refs.h" #include "fmt-merge-msg.h" #include "commit.h" +#include "object-store.h" int trust_executable_bit =1 ;int trust_ctime =1 ;
@@ -244,9 +245,9 @@ const char *get_git_work_tree(void)char *get_object_directory (void ) { -if (!the_repository->objectdir) +if (!the_repository->objects.objectdir)BUG ("git environment hasn't been setup" ); -return the_repository->objectdir; +return the_repository->objects.objectdir; }int odb_mkstemp (struct strbuf *template,const char *pattern)
object-store.h new file +15 -0
@@ -0,0 +1,15 @@ +#ifndef OBJECT_STORE_H +#define OBJECT_STORE_H + +struct raw_object_store { +/* +* Path to the repository's object store. +* Cannot be NULL after initialization. +*/ +char *objectdir; +}; +#define RAW_OBJECT_STORE_INIT { NULL } + +void raw_object_store_clear (struct raw_object_store *o); + +#endif /* OBJECT_STORE_H */
object.c +5 -0
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags) obj->flags &= ~flags; } } + +void raw_object_store_clear (struct raw_object_store *o) +{ +free (o->objectdir); +}
path.c +1 -1
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,strbuf_splice (buf,0 , buf->len, repo->index_file,strlen (repo->index_file)); else if (dir_prefix(base, "objects")) - replace_dir(buf, git_dir_len + 7, repo->objectdir); + replace_dir(buf, git_dir_len + 7, repo->objects.objectdir); else if (git_hooks_path && dir_prefix(base, "hooks")) replace_dir(buf, git_dir_len + 5, git_hooks_path); else if (repo->different_commondir)
repository.c +13 -4
@@ -1,11 +1,18 @@#include "cache.h" #include "repository.h" +#include "object-store.h" #include "config.h" #include "submodule-config.h" /* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL , +RAW_OBJECT_STORE_INIT , +NULL ,NULL ,NULL , +NULL ,NULL ,NULL , + &the_index, + &hash_algos[GIT_HASH_SHA1 ], +0 ,0 };struct repository *the_repository = &the_repo;
@@ -42,8 +49,8 @@ static void repo_setup_env(struct repository *repo) !repo->ignore_env);free (repo->commondir); repo->commondir =strbuf_detach (&sb,NULL ); -free (repo->objectdir); - repo->objectdir =git_path_from_env (DB_ENVIRONMENT , repo->commondir, +free (repo->objects.objectdir); + repo->objects.objectdir =git_path_from_env (DB_ENVIRONMENT , repo->commondir,"objects" , !repo->ignore_env);free (repo->graft_file); repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
@@ -209,12 +216,14 @@ void repo_clear(struct repository *repo) {FREE_AND_NULL (repo->gitdir);FREE_AND_NULL (repo->commondir); -FREE_AND_NULL (repo->objectdir);FREE_AND_NULL (repo->graft_file);FREE_AND_NULL (repo->index_file);FREE_AND_NULL (repo->worktree);FREE_AND_NULL (repo->submodule_prefix); +raw_object_store_clear (&repo->objects); +memset (&repo->objects,0 ,sizeof (repo->objects)); +if (repo->config) {git_configset_clear (repo->config);FREE_AND_NULL (repo->config);
repository.h +4 -3
@@ -1,6 +1,8 @@#ifndef REPOSITORY_H #define REPOSITORY_H +#include "object-store.h" +struct config_set ;struct index_state ;struct submodule_cache ;
@@ -21,10 +23,9 @@ struct repository {char *commondir;/* -* Path to the repository's object store. -* Cannot be NULL after initialization. +* Holds any information related to the object store. */ -char *objectdir; +struct raw_object_store objects;/* * Path to the repository's graft file.
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 01/27] repository: introduce raw object store field
Hi,
Quoted 16 lines · Stefan Beller
The raw object store field will contain any objects needed for access to objects in a given repository.
This patch introduces the raw object store and populates it with the
objectdir, which used to be part of the repository struct.As the struct gains members, we'll also populate the function to clear the memory for these members.
In a later we'll introduce a struct
object_parser, that will complement the object parsing in a repository struct: The raw object parser is the layer that will provide access to raw object content, while the higher level object parser code will parse raw objects and keeps track of parenthood and other object relationships using 'struct object'. For now only add the lower level to the repository struct.
Quoted 3 lines
Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>---
Heh, I suppose that sign-off makes me not a great candidate for reviewing this. It's been long enough since I looked at the patch that I feel okay trying anyway.
[...]
Quoted 4 lines
object-store.h new file +1
-0@@ -0,0 +1,15 @@ +#ifndef OBJECT_STORE_H
object-store.h line 11
Quoted 10 lines
object-store.h new file +10
-0+#define OBJECT_STORE_H + +struct raw_object_store { +/* +* Path to the repository's object store. +* Cannot be NULL after initialization. +*/ +char *objectdir; +}; +#define RAW_OBJECT_STORE_INIT { NULL }
Who owns and is responsible for freeing objectdir?
object-store.h line 13
Quoted 2 lines
object-store.h new file +2
-0+ +void raw_object_store_clear (struct raw_object_store *o);
Oh, that answers that.
It could be worth a note in the doc comment anyway, but I don't mind not having one.
[...]
object-store.h
Quoted 5 lines
object-store.h new file +5
-0+ +void raw_object_store_clear (struct raw_object_store *o) +{ +free (o->objectdir); +}
Should this use FREE_AND_NULL?
[...]
Quoted 2 lines
repository.c +0
-0
[...]
repository.c static void repo_setup_env(struct repository *repo) · line 52
Quoted 7 lines
repository.c +1
-2@@ -42,8 +49,8 @@ static void repo_setup_env(struct repository *repo) !repo->ignore_env);free (repo->commondir); repo->commondir =strbuf_detach (&sb,NULL ); -free (repo->objectdir); - repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir, +free (repo->objects.objectdir);
Should this call raw_object_store_clear instead of calling free directly?
repository.c static void repo_setup_env(struct repository *repo) · line 54
Quoted 2 lines
repository.c +1
-0+ repo->objects.objectdir =git_path_from_env (DB_ENVIRONMENT , repo->commondir,"objects" , !repo->ignore_env);
Long line. One way to break it would be
repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir, "objects", !repo->ignore_env);
Quoted 4 lines
repository.c +0
-0free (repo->graft_file); repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,@@ -209,12 +216,14 @@ void repo_clear(struct repository *repo) {
repository.c void repo_clear(struct repository *repo) · line 225
Quoted 10 lines
repository.c +2
-1FREE_AND_NULL (repo->gitdir);FREE_AND_NULL (repo->commondir); -FREE_AND_NULL (repo->objectdir);FREE_AND_NULL (repo->graft_file);FREE_AND_NULL (repo->index_file);FREE_AND_NULL (repo->worktree);FREE_AND_NULL (repo->submodule_prefix); +raw_object_store_clear (&repo->objects); +memset (&repo->objects,0 ,sizeof (repo->objects));
If raw_object_store_clear uses FREE_AND_NULL, this memset won't be necessary.
[...]
Quoted 12 lines
repository.h +2
-0@@ -1,6 +1,8 @@#ifndef REPOSITORY_H #define REPOSITORY_H +#include "object-store.h" +struct config_set ;struct index_state ;struct submodule_cache ;
repository.h struct repository { · line 27
Quoted 7 lines
repository.h +1
-2@@ -21,10 +23,9 @@ struct repository {char *commondir;/* -* Path to the repository's object store. -* Cannot be NULL after initialization. +* Holds any information related to the object store. */
This comment doesn't make it clear to me what the field represents. Can it be replaced with some of the description from the commit message?
repository.h struct repository { · line 29
Quoted 3 lines
repository.h +1
-1-char *objectdir; +struct raw_object_store objects;
Thanks,
Jonathan
PATCH v3 2/27 object-store: migrate alternates struct and functions from cache.h
In reply to Stefan BellerMigrate the struct alternate_object_database and all its related functions to the object store as these functions are easier found in that header. The migration is just a verbatim copy, no need to include the object store header at any C file, because cache.h includes repository.h which in turn includes the object-store.h
--- cache.h | 51 -------------------------------------------------- object-store.h | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 51 deletions(-)
cache.h +0 -51
@@ -1576,57 +1576,6 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);extern void schedule_dir_for_removal (const char *name,int len);extern void remove_scheduled_dirs (void ); -extern struct alternate_object_database { -struct alternate_object_database *next; - -/* see alt_scratch_buf() */ -struct strbuf scratch; -size_t base_len; - -/* -* Used to store the results of readdir(3) calls when searching -* for unique abbreviated hashes. This cache is never -* invalidated, thus it's racy and not necessarily accurate. -* That's fine for its purpose; don't use it for tasks requiring -* greater accuracy! -*/ -char loose_objects_subdir_seen[256 ]; -struct oid_array loose_objects_cache; - -char path[FLEX_ARRAY ]; -} *alt_odb_list; -extern void prepare_alt_odb (void ); -extern char *compute_alternate_path (const char *path,struct strbuf *err); -typedef int alt_odb_fn (struct alternate_object_database *,void *); -extern int foreach_alt_odb (alt_odb_fn ,void *); - -/* -* Allocate a "struct alternate_object_database" but do _not_ actually -* add it to the list of alternates. -*/ -struct alternate_object_database *alloc_alt_odb (const char *dir); - -/* -* Add the directory to the on-disk alternates file; the new entry will also -* take effect in the current process. -*/ -extern void add_to_alternates_file (const char *dir); - -/* -* Add the directory to the in-memory list of alternates (along with any -* recursive alternates it points to), but do not modify the on-disk alternates -* file. -*/ -extern void add_to_alternates_memory (const char *dir); - -/* -* Returns a scratch strbuf pre-filled with the alternate object directory, -* including a trailing slash, which can be used to access paths in the -* alternate. Always use this over direct access to alt->scratch, as it -* cleans up any previous use of the scratch buffer. -*/ -extern struct strbuf *alt_scratch_buf (struct alternate_object_database *alt); -struct pack_window {struct pack_window *next;unsigned char *base;
object-store.h +51 -0
57 lines not on this page · raw message
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 02/27] object-store: migrate alternates struct and functions from cache.h
Quoted 5 lines · Stefan Beller
Migrate the struct
alternate_object_databaseand all its related functions to the object store as these functions are easier found in that header. The migration is just a verbatim copy, no need to include the object store header at any C file, becausecache.hincludesrepository.hwhich in turn includes theobject-store.h
Nice. Always love seeing cache.h get smaller
Signature · 1 line
Brandon Williams
PATCH v3 3/27 object-store: move alt_odb_list and alt_odb_tail to object store
In reply to Stefan BellerIn a process with multiple repositories open, alternates should be associated to a single repository and not shared globally. Move alt_odb_list and alt_odb_tail into the_repository and adjust callers to reflect this.
Now that the alternative object data base is per repository, we're leaking its memory upon freeing a repository. The next patch plugs this hole.
No functional change intended.
--- builtin/fsck.c | 4 +++- object-store.h | 9 ++++++--- packfile.c | 3 ++- sha1_file.c | 25 ++++++++++++------------- sha1_name.c | 3 ++- 5 files changed, 25 insertions(+), 19 deletions(-)
builtin/fsck.c +3 -1
@@ -1,5 +1,6 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "commit.h" #include "tree.h"
@@ -716,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)fsck_object_dir (get_object_directory ());prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; + alt; alt = alt->next)fsck_object_dir (alt->path);if (check_full) {
object-store.h +6 -3
@@ -1,7 +1,7 @@ #ifndef OBJECT_STORE_H #define OBJECT_STORE_H -extern struct alternate_object_database { +struct alternate_object_database { struct alternate_object_database *next; /* see alt_scratch_buf() */
@@ -19,7 +19,7 @@ extern struct alternate_object_database {struct oid_array loose_objects_cache;char path[FLEX_ARRAY ]; -} *alt_odb_list; +};void prepare_alt_odb (void );char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);
@@ -58,8 +58,11 @@ struct raw_object_store {* Cannot be NULL after initialization. */ char *objectdir; + +struct alternate_object_database *alt_odb_list; +struct alternate_object_database **alt_odb_tail; }; -#define RAW_OBJECT_STORE_INIT { NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +2 -1
@@ -1,6 +1,7 @@#include "cache.h" #include "list.h" #include "pack.h" +#include "repository.h" #include "dir.h" #include "mergesort.h" #include "packfile.h"
@@ -891,7 +892,7 @@ void prepare_packed_git(void)return ;prepare_packed_git_one (get_object_directory (),1 );prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();prepare_packed_git_mru ();
sha1_file.c +12 -13
@@ -22,6 +22,7 @@#include "pack-revindex.h" #include "sha1-lookup.h" #include "bulk-checkin.h" +#include "repository.h" #include "streaming.h" #include "dir.h" #include "list.h"
@@ -342,9 +343,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,return buf->buf; } -struct alternate_object_database *alt_odb_list; -static struct alternate_object_database **alt_odb_tail; -/* * Return non-zero iff the path is usable as an alternate object database. */
@@ -364,7 +362,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)* Prevent the common mistake of listing the same * thing twice, or object directory itself. */ -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {if (!fspathcmp (path->buf, alt->path))return 0 ; }
@@ -424,8 +422,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base, ent =alloc_alt_odb (pathbuf.buf);/* add the alternate entry */ - *alt_odb_tail = ent; - alt_odb_tail = &(ent->next); + *the_repository->objects.alt_odb_tail = ent; + the_repository->objects.alt_odb_tail = &(ent->next); ent->next =NULL ;/* recursively add alternates */
@@ -559,7 +557,7 @@ void add_to_alternates_file(const char *reference)fprintf_or_die (out,"%s\n" , reference);if (commit_lock_file (&lock))die_errno ("unable to move new alternates file into place" ); -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)link_alt_odb_entries (reference,'\n' ,NULL ,0 ); }free (alts);
@@ -657,7 +655,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)int r =0 ;prepare_alt_odb (); -for (ent = alt_odb_list; ent; ent = ent->next) { +for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) { r =fn (ent, cb);if (r)break ;
@@ -669,12 +667,13 @@ void prepare_alt_odb(void) {const char *alt; -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)return ; alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - alt_odb_tail = &alt_odb_list; + the_repository->objects.alt_odb_tail = + &the_repository->objects.alt_odb_list;link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 );read_info_alternates (get_object_directory (),0 );
@@ -718,7 +717,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen) {struct alternate_object_database *alt;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {const char *path =alt_sha1_path (alt, sha1);if (check_and_freshen_file (path, freshen))return 1 ;
@@ -878,7 +877,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,prepare_alt_odb (); errno =ENOENT ; -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);if (!lstat (*path, st))return 0 ;
@@ -908,7 +907,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path) most_interesting_errno = errno;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);if (fd >=0 )
sha1_name.c +2 -1
@@ -10,6 +10,7 @@#include "dir.h" #include "sha1-array.h" #include "packfile.h" +#include "repository.h" static int get_oid_oneline (const char *,struct object_id *,struct commit_list *);
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)*/ fakeent =alloc_alt_odb (get_object_directory ()); } - fakeent->next = alt_odb_list; + fakeent->next = the_repository->objects.alt_odb_list;for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {int pos;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 03/27] object-store: move alt_odb_list and alt_odb_tail to object store
Quoted 5 lines · Stefan Beller
In a process with multiple repositories open, alternates should be associated to a single repository and not shared globally. Move
alt_odb_listandalt_odb_tailintothe_repositoryand adjust callers to reflect this.
Quoted 8 lines
Now that the alternative object data base is per repository, we're leaking its memory upon freeing a repository. The next patch plugs this hole.
No functional change intended.
Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>
Looks good aside from the one spacing issue (though it's not that big of a deal).
Quoted 20 lines
--- builtin/fsck.c | 4 +++- object-store.h | 9 ++++++--- packfile.c | 3 ++- sha1_file.c | 25 ++++++++++++------------- sha1_name.c | 3 ++- 5 files changed, 25 insertions(+), 19 deletions(-)builtin/fsck.c +1
-0@@ -1,5 +1,6 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "commit.h" #include "tree.h"
builtin/fsck.c int cmd_fsck(int argc, const char **argv, const char *prefix) · line 721
Quoted 6 lines
builtin/fsck.c +2
-1@@ -716,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)fsck_object_dir (get_object_directory ());prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; + alt; alt = alt->next)
Indentation on this line looks odd.
Signature · 1 line
Brandon Williams
PATCH v3 4/27 object-store: free alt_odb_list
In reply to Stefan BellerFree the memory and reset alt_odb_{list, tail} to NULL.
--- object.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
object.c +17 -0
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags) } } +static void free_alt_odb (struct alternate_object_database *alt) +{ +strbuf_release (&alt->scratch); +oid_array_clear (&alt->loose_objects_cache); +} + +static void free_alt_odbs (struct raw_object_store *o) +{ +while (o->alt_odb_list) { +free_alt_odb (o->alt_odb_list); + o->alt_odb_list = o->alt_odb_list->next; + } +} +void raw_object_store_clear (struct raw_object_store *o) {free (o->objectdir); + +free_alt_odbs (o); + o->alt_odb_tail =NULL ; }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 04/27] object-store: free alt_odb_list
Quoted 1 line · Stefan Beller
Free the memory and reset alt_odb_{list, tail} to NULL.
Good to see memory leaks being avoided (well they will be on other repository objects)
Quoted 38 lines
Signed-off-by Stefan Beller <[email protected]>--- object.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)object.c +17
-0@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags) } } +static void free_alt_odb (struct alternate_object_database *alt) +{ +strbuf_release (&alt->scratch); +oid_array_clear (&alt->loose_objects_cache); +} + +static void free_alt_odbs (struct raw_object_store *o) +{ +while (o->alt_odb_list) { +free_alt_odb (o->alt_odb_list); + o->alt_odb_list = o->alt_odb_list->next; + } +} +void raw_object_store_clear (struct raw_object_store *o) {free (o->objectdir); + +free_alt_odbs (o); + o->alt_odb_tail =NULL ; }Signature · 1 line
2.16.1.291.g4437f3f132-goog
Signature · 1 line
Brandon Williams