Moving global state into the repository object (part 1)
PATCH v3 5/27 object-store: move packed_git and packed_git_mru to object store
In reply to Stefan BellerIn a process with multiple repositories open, packfile accessors should be associated to a single repository and not shared globally. Move packed_git and packed_git_mru into the_repository and adjust callers to reflect this.
Patch generated by
1. Moving the struct packed_git declaration to object-store.h and packed_git, packed_git_mru globals to struct object_store.
2. Applying the semantic patch contrib/coccinelle/refactoring/packed_git.cocci to adjust callers. This semantic patch is placed in a sub directory of the coccinelle contrib dir, as this semantic patch is not expected to be of general usefulness; it is only useful during developing this series and merging it with other topics in flight. At a later date, just delete that semantic patch.
3. Applying line wrapping fixes from "make style" to break the resulting long lines.
4. Adding missing #includes of repository.h and object-store.h where needed.
5. As the packfiles are now owned by an objectstore/repository, which is ephemeral unlike globals, we introduce memory leaks. So address them in raw_object_store_clear().
--- builtin/count-objects.c | 6 ++++-- builtin/fsck.c | 7 +++++-- builtin/gc.c | 4 +++- builtin/index-pack.c | 1 + builtin/pack-objects.c | 20 +++++++++++--------- builtin/pack-redundant.c | 6 ++++-- builtin/receive-pack.c | 1 + cache.h | 29 ---------------------------- fast-import.c | 6 ++++-- http-backend.c | 6 ++++-- http-push.c | 1 + http-walker.c | 1 + http.c | 1 + object-store.h | 41 +++++++++++++++++++++++++++++++++++++++- object.c | 7 +++++++ pack-bitmap.c | 4 +++- pack-check.c | 1 + pack-revindex.c | 1 + packfile.c | 39 +++++++++++++++++++------------------- reachable.c | 1 + repository.c | 2 ++ server-info.c | 6 ++++-- sha1_name.c | 5 +++-- 23 files changed, 122 insertions(+), 74 deletions(-)
builtin/count-objects.c +4 -2
@@ -7,6 +7,8 @@#include "cache.h" #include "config.h" #include "dir.h" +#include "repository.h" +#include "object-store.h" #include "builtin.h" #include "parse-options.h" #include "quote.h"
@@ -120,9 +122,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)struct strbuf loose_buf =STRBUF_INIT ;struct strbuf pack_buf =STRBUF_INIT ;struct strbuf garbage_buf =STRBUF_INIT ; -if (!packed_git) +if (!the_repository->objects.packed_git)prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ;if (open_pack_index (p))
builtin/fsck.c +5 -2
@@ -2,6 +2,7 @@#include "cache.h" #include "repository.h" #include "config.h" +#include "object-store.h" #include "commit.h" #include "tree.h" #include "blob.h"
@@ -729,7 +730,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)prepare_packed_git ();if (show_progress) { -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {if (open_pack_index (p))continue ; total += p->num_objects;
@@ -737,7 +739,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) progress =start_progress (_ ("Checking objects" ), total); } -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {/* verify gives error messages itself */ if (verify_pack (p, fsck_obj_buffer, progress, count))
builtin/gc.c +3 -1
@@ -11,6 +11,7 @@*/ #include "builtin.h" +#include "repository.h" #include "config.h" #include "tempfile.h" #include "lockfile.h"
@@ -18,6 +19,7 @@#include "run-command.h" #include "sigchain.h" #include "argv-array.h" +#include "object-store.h" #include "commit.h" #include "packfile.h"
@@ -173,7 +175,7 @@ static int too_many_packs(void)return 0 ;prepare_packed_git (); -for (cnt =0 , p = packed_git; p; p = p->next) { +for (cnt =0 , p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ;if (p->pack_keep)
builtin/index-pack.c +1 -0
@@ -12,6 +12,7 @@#include "exec_cmd.h" #include "streaming.h" #include "thread-utils.h" +#include "object-store.h" #include "packfile.h" static const char index_pack_usage[] =
builtin/pack-objects.c +11 -9
@@ -1,6 +1,8 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" +#include "object-store.h" #include "attr.h" #include "object.h" #include "blob.h"
@@ -1025,8 +1027,7 @@ static int want_object_in_pack(const struct object_id *oid,if (want !=-1 )return want; } - -list_for_each (pos, &packed_git_mru) { +list_for_each (pos, &the_repository->objects.packed_git_mru) { struct packed_git *p = list_entry(pos, struct packed_git, mru);off_t offset;
@@ -1044,7 +1045,7 @@ static int want_object_in_pack(const struct object_id *oid, } want =want_found_object (exclude, p);if (!exclude && want >0 ) -list_move (&p->mru, &packed_git_mru); +list_move (&p->mru, &the_repository->objects.packed_git_mru);if (want !=-1 )return want; }
@@ -2673,7 +2674,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)memset (&in_pack,0 ,sizeof (in_pack)); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {struct object_id oid;struct object *o;
@@ -2736,7 +2737,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)static struct packed_git *last_found = (void *)1 ;struct packed_git *p; - p = (last_found != (void *)1 ) ? last_found : packed_git; + p = (last_found != (void *)1 ) ? last_found : + the_repository->objects.packed_git;while (p) { if ((!p->pack_local || p->pack_keep) &&
@@ -2745,7 +2747,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)return 1 ; }if (p == last_found) - p = packed_git; + p = the_repository->objects.packed_git;else p = p->next;if (p == last_found)
@@ -2781,7 +2783,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)uint32_t i;struct object_id oid; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local || p->pack_keep)continue ;
@@ -3152,7 +3154,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)prepare_packed_git ();if (ignore_packed_keep) {struct packed_git *p; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)if (p->pack_local && p->pack_keep)break ;if (!p)/* no keep-able packs found */
@@ -3165,7 +3167,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)* also covers non-local objects */ struct packed_git *p; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local) { have_non_local_packs =1 ;break ;
builtin/pack-redundant.c +4 -2
@@ -7,6 +7,8 @@*/ #include "builtin.h" +#include "repository.h" +#include "object-store.h" #include "packfile.h" #define BLKSIZE 512
@@ -571,7 +573,7 @@ static struct pack_list * add_pack(struct packed_git *p)static struct pack_list *add_pack_file (const char *filename) { -struct packed_git *p = packed_git; +struct packed_git *p = the_repository->objects.packed_git;if (strlen (filename) <40 )die ("Bad pack filename: %s" , filename);
@@ -586,7 +588,7 @@ static struct pack_list * add_pack_file(const char *filename)static void load_all (void ) { -struct packed_git *p = packed_git; +struct packed_git *p = the_repository->objects.packed_git;while (p) {add_pack (p);
builtin/receive-pack.c +1 -0
@@ -7,6 +7,7 @@#include "sideband.h" #include "run-command.h" #include "exec_cmd.h" +#include "object-store.h" #include "commit.h" #include "object.h" #include "remote.h"
cache.h +0 -29
@@ -1585,35 +1585,6 @@ struct pack_window {unsigned int inuse_cnt; }; -extern struct packed_git { -struct packed_git *next; -struct list_head mru; -struct pack_window *windows; -off_t pack_size; -const void *index_data; -size_t index_size; -uint32_t num_objects; -uint32_t num_bad_objects; -unsigned char *bad_object_sha1; -int index_version; -time_t mtime; -int pack_fd; -unsigned pack_local:1 , - pack_keep:1 , - freshened:1 , - do_not_close:1 , - pack_promisor:1 ; -unsigned char sha1[20 ]; -struct revindex_entry *revindex; -/* something like ".git/objects/pack/xxxxx.pack" */ -char pack_name[FLEX_ARRAY ];/* more */ -} *packed_git; - -/* -* A most-recently-used ordered version of the packed_git list. -*/ -extern struct list_head packed_git_mru; -struct pack_entry {off_t offset;unsigned char sha1[20 ];
fast-import.c +4 -2
@@ -154,8 +154,10 @@ Format of STDIN stream:#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "lockfile.h" +#include "object-store.h" #include "object.h" #include "blob.h" #include "tree.h"
@@ -1110,7 +1112,7 @@ static int store_object(if (e->idx.offset) { duplicate_count_by_type[type]++;return 1 ; - }else if (find_sha1_pack (oid.hash, packed_git)) { + }else if (find_sha1_pack (oid.hash, the_repository->objects.packed_git)) { e->type = type; e->pack_id =MAX_PACK_ID ; e->idx.offset =1 ;/* just not zero! */
@@ -1305,7 +1307,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark) duplicate_count_by_type[OBJ_BLOB ]++;truncate_pack (&checkpoint); - }else if (find_sha1_pack (oid.hash, packed_git)) { + }else if (find_sha1_pack (oid.hash, the_repository->objects.packed_git)) { e->type =OBJ_BLOB ; e->pack_id =MAX_PACK_ID ; e->idx.offset =1 ;/* just not zero! */
http-backend.c +4 -2
@@ -1,7 +1,9 @@#include "cache.h" #include "config.h" +#include "repository.h" #include "refs.h" #include "pkt-line.h" +#include "object-store.h" #include "object.h" #include "tag.h" #include "exec_cmd.h"
@@ -518,13 +520,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)select_getanyfile (hdr);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_local) cnt++; }strbuf_grow (&buf, cnt *53 +2 ); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_local)strbuf_addf (&buf,"P %s\n" , p->pack_name + objdirlen +6 ); }
http-push.c +1 -0
@@ -1,4 +1,5 @@#include "cache.h" +#include "object-store.h" #include "commit.h" #include "tag.h" #include "blob.h"
http-walker.c +1 -0
@@ -4,6 +4,7 @@#include "http.h" #include "list.h" #include "transport.h" +#include "object-store.h" #include "packfile.h" struct alt_base {
http.c +1 -0
@@ -1,6 +1,7 @@#include "git-compat-util.h" #include "http.h" #include "config.h" +#include "object-store.h" #include "pack.h" #include "sideband.h" #include "run-command.h"
object-store.h +40 -1
56 lines not on this page · raw message
object.c +7 -0
@@ -466,4 +466,11 @@ void raw_object_store_clear(struct raw_object_store *o)free_alt_odbs (o); o->alt_odb_tail =NULL ; + +while (!list_empty (&o->packed_git_mru)) +list_del (&o->packed_git_mru); +/* +* TODO: call close_all_packs once migrated to +* take an object store argument +*/ }
pack-bitmap.c +3 -1
@@ -5,11 +5,13 @@#include "revision.h" #include "progress.h" #include "list-objects.h" +#include "object-store.h" #include "pack.h" #include "pack-bitmap.h" #include "pack-revindex.h" #include "pack-objects.h" #include "packfile.h" +#include "repository.h" /* * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +337,7 @@ static int open_pack_bitmap(void)assert (!bitmap_git.map && !bitmap_git.loaded);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (open_pack_bitmap_1 (p) ==0 ) ret =0 ; }
pack-check.c +1 -0
@@ -1,5 +1,6 @@#include "cache.h" #include "pack.h" +#include "object-store.h" #include "pack-revindex.h" #include "progress.h" #include "packfile.h"
pack-revindex.c +1 -0
@@ -1,5 +1,6 @@#include "cache.h" #include "pack-revindex.h" +#include "object-store.h" /* * Pack index for existing packs give us easy access to the offsets into
packfile.c +20 -19
@@ -7,6 +7,7 @@#include "packfile.h" #include "delta.h" #include "list.h" +#include "object-store.h" #include "streaming.h" #include "sha1-lookup.h" #include "commit.h"
@@ -45,8 +46,6 @@ static unsigned int pack_open_fds;static unsigned int pack_max_fds;static size_t peak_pack_mapped;static size_t pack_mapped; -struct packed_git *packed_git; -LIST_HEAD (packed_git_mru);#define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; }
@@ -246,7 +245,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -316,7 +315,7 @@ void close_all_packs(void) {struct packed_git *p; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)if (p->do_not_close)die ("BUG: want to close pack marked 'do-not-close'" );else
@@ -384,7 +383,7 @@ static int close_one_pack(void)struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -686,8 +685,8 @@ void install_packed_git(struct packed_git *pack)if (pack->pack_fd !=-1 ) pack_open_fds++; - pack->next = packed_git; - packed_git = pack; + pack->next = the_repository->objects.packed_git; + the_repository->objects.packed_git = pack; }void (*report_garbage)(unsigned seen_bits,const char *path);
@@ -769,7 +768,8 @@ static void prepare_packed_git_one(char *objdir, int local) base_len = path.len;if (strip_suffix_mem (path.buf, &base_len,".idx" )) {/* Don't reopen a pack we already have. */ -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {size_t len; if (strip_suffix(p->pack_name, ".pack", &len) && len == base_len &&
@@ -820,7 +820,7 @@ unsigned long approximate_object_count(void)prepare_packed_git (); count =0 ; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (open_pack_index (p))continue ; count += p->num_objects;
@@ -869,18 +869,19 @@ static int sort_pack(const void *a_, const void *b_)static void rearrange_packed_git (void ) { - packed_git =llist_mergesort (packed_git, get_next_packed_git, - set_next_packed_git, sort_pack); + the_repository->objects.packed_git =llist_mergesort ( + the_repository->objects.packed_git, get_next_packed_git, + set_next_packed_git, sort_pack); }static void prepare_packed_git_mru (void ) {struct packed_git *p; -INIT_LIST_HEAD (&packed_git_mru); +INIT_LIST_HEAD (&the_repository->objects.packed_git_mru); -for (p = packed_git; p; p = p->next) -list_add_tail (&p->mru, &packed_git_mru); +for (p = the_repository->objects.packed_git; p; p = p->next) +list_add_tail (&p->mru, &the_repository->objects.packed_git_mru); }static int prepare_packed_git_run_once =0 ;
@@ -1014,7 +1015,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)struct packed_git *p;unsigned i; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)for (i =0 ; i < p->num_bad_objects; i++)if (!hashcmp (sha1, p->bad_object_sha1 +20 * i))return p;
@@ -1845,13 +1846,13 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)struct list_head *pos;prepare_packed_git (); -if (!packed_git) +if (!the_repository->objects.packed_git)return 0 ; -list_for_each (pos, &packed_git_mru) { +list_for_each (pos, &the_repository->objects.packed_git_mru) { struct packed_git *p = list_entry(pos, struct packed_git, mru);if (fill_pack_entry (sha1, e, p)) { -list_move (&p->mru, &packed_git_mru); +list_move (&p->mru, &the_repository->objects.packed_git_mru);return 1 ; } }
@@ -1898,7 +1899,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)int pack_errors =0 ;prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ; if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
reachable.c +1 -0
@@ -1,5 +1,6 @@#include "cache.h" #include "refs.h" +#include "object-store.h" #include "tag.h" #include "commit.h" #include "blob.h"
repository.c +2 -0
@@ -142,6 +142,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree) repo->ignore_env =1 ; +INIT_LIST_HEAD (&repo->objects.packed_git_mru); +if (repo_init_gitdir (repo, gitdir)) goto error;
server-info.c +4 -2
@@ -1,5 +1,7 @@#include "cache.h" +#include "repository.h" #include "refs.h" +#include "object-store.h" #include "object.h" #include "commit.h" #include "tag.h"
@@ -200,7 +202,7 @@ static void init_pack_info(const char *infofile, int force) objdirlen =strlen (objdir);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {/* we ignore things on alternate path since they are * not available to the pullers in general. */
@@ -210,7 +212,7 @@ static void init_pack_info(const char *infofile, int force) } num_pack = i; info =xcalloc (num_pack,sizeof (struct pack_info *)); -for (i =0 , p = packed_git; p; p = p->next) { +for (i =0 , p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ; info[i] =xcalloc (1 ,sizeof (struct pack_info ));
sha1_name.c +3 -2
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)struct packed_git *p;prepare_packed_git (); -for (p = packed_git; p && !ds->ambiguous; p = p->next) +for (p = the_repository->objects.packed_git; p && !ds->ambiguous; + p = p->next)unique_in_pack (p, ds); }
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)struct packed_git *p;prepare_packed_git (); -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)find_abbrev_len_for_pack (p, mad); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store
object-store.h struct raw_object_store { · line 104
Quoted 9 lines · Stefan Beller
object-store.h +9
-0+ +/* +* The mru list_head is supposed to be initialized using +* the LIST_HEAD macro, assigning prev/next to itself. +* However this doesn't work in this case as some compilers dislike +* that macro on member variables. Use NULL instead as that is defined +* and accepted, deferring the real init to prepare_packed_git_mru(). */ +#define __MRU_INIT { NULL, NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
I do not think it has to be this way to abuse two NULLs, if you faithfully mimicked how LIST_HEAD() macro is constructed. The reason why it does not try to introduce
struct list_head x = LIST_HEAD_INIT;
and instead, uses
LIST_HEAD(x);
is because of the need for self referral. If we learn from it, we can do the same, i.e. instead of doing
struct raw_object_store x = RAW_OBJECT_STORE_INIT;
we can do
RAW_OBJECT_STORE(x);
that expands to
struct raw_object_store x = {
...
{ &x.packed_git_mru, &x.packed_git_mru },
...
};no? Then we do not need such a lengthy comment that reads only like an excuse ;-)
Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store
Quoted 33 lines · Junio C Hamano
object-store.h struct raw_object_store { · line 104
Quoted 9 lines · Stefan Beller
object-store.h +9
-0+ +/* +* The mru list_head is supposed to be initialized using +* the LIST_HEAD macro, assigning prev/next to itself. +* However this doesn't work in this case as some compilers dislike +* that macro on member variables. Use NULL instead as that is defined +* and accepted, deferring the real init to prepare_packed_git_mru(). */ +#define __MRU_INIT { NULL, NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }I do not think it has to be this way to abuse two NULLs, if you faithfully mimicked how
LIST_HEAD()macro is constructed. The reason why it does not try to introducestruct list_head x = LIST_HEAD_INIT;and instead, uses
LIST_HEAD(x);is because of the need for self referral. If we learn from it, we can do the same, i.e. instead of doing
struct raw_object_store x = RAW_OBJECT_STORE_INIT;we can do
RAW_OBJECT_STORE(x);that expands to
Quoted 8 lines
struct raw_object_store x = { ... { &x.packed_git_mru, &x.packed_git_mru }, ... };no? Then we do not need such a lengthy comment that reads only like an excuse ;-)
We cannot do this, because the object store is directly embedded into the the_repository (not as a pointer), which is a global on its own. So we'd have to do this at the repository level, which I would not want. I'd want to have the #defines where the structs are declared.
Any guidance on how to do that correctly with self referential definitions without making the object store a pointer then?
Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store
In reply to Stefan BellerHi,
Quoted 10 lines · Stefan Beller
In a process with multiple repositories open, packfile accessors should be associated to a single repository and not shared globally. Move
packed_gitandpacked_git_mruintothe_repositoryand adjust callers to reflect this.Patch generated by
1. Moving the struct
packed_gitdeclaration toobject-store.handpacked_git,packed_git_mruglobals to structobject_store.
Quoted 7 lines
2. Applying the semantic patch
contrib/coccinelle/refactoring/packed_git.coccito adjust callers. This semantic patch is placed in a sub directory of the coccinelle contrib dir, as this semantic patch is not expected to be of general usefulness; it is only useful during developing this series and merging it with other topics in flight. At a later date, just delete that semantic patch.
Can the semantic patch go in the commit message instead? It is very brief.
Actually, I don't see this semantic patch in the diffstat. Is the commit message stale?
Quoted 5 lines
3. Applying line wrapping fixes from "make style" to break the resulting long lines.
4. Adding missing #includes of
repository.handobject-store.hwhere needed.
Is there a way to automate this step? (I'm asking for my own reference when writing future patches, not because of any concern about the correctness of this one.)
Quoted 4 lines
5. As the packfiles are now owned by an objectstore/repository, which is ephemeral unlike globals, we introduce memory leaks. So address them in
raw_object_store_clear().
The compound words are confusing me. What is an objectstore/repository? Are these referring to particular identifiers or something else?
Would some wording like the following work?
5. Freeing packed_git and packed_git_mru in raw_object_store_clear to avoid a per-repository memory leak. Previously they were global singletons, so code to free them did not exist.
[...]
builtin/index-pack.c line 18
Quoted 10 lines
builtin/index-pack.c +1
-0@@ -12,6 +12,7 @@#include "exec_cmd.h" #include "streaming.h" #include "thread-utils.h" +#include "object-store.h" #include "packfile.h" static const char index_pack_usage[] =
Change from a different patch leaked into this one?
[...]
Quoted 1 line
+++
b/builtin/pack-objects.c
[...]
builtin/index-pack.c static int want_object_in_pack(const struct object_id *oid, · line 1048
Quoted 6 lines
builtin/index-pack.c +1
-1@@ -1044,7 +1045,7 @@ static int want_object_in_pack(const struct object_id *oid, } want =want_found_object (exclude, p);if (!exclude && want >0 ) -list_move (&p->mru, &packed_git_mru); +list_move (&p->mru, &the_repository->objects.packed_git_mru);
Long line. Can "make style" catch this?
[...]
builtin/index-pack.c line 13
Quoted 9 lines
builtin/index-pack.c +2
-0+++ b/builtin/receive-pack.c@@ -7,6 +7,7 @@#include "sideband.h" #include "run-command.h" #include "exec_cmd.h" +#include "object-store.h" #include "commit.h" #include "object.h" #include "remote.h"
Another change leaked in?
[...]
Quoted 7 lines
cache.h +0
-1@@ -1585,35 +1585,6 @@ struct pack_window {unsigned int inuse_cnt; }; -extern struct packed_git {
[...]
Quoted 1 line
-}
*packed_git;
Move detecting diff confirms that this wasn't modified. Thanks for creating it.
[...]
Quoted 1 line
+++
b/fast-import.c
[...]
cache.h static int store_object( · line 1115
Quoted 6 lines
cache.h +1
-1@@ -1110,7 +1112,7 @@ static int store_object(if (e->idx.offset) { duplicate_count_by_type[type]++;return 1 ; - }else if (find_sha1_pack (oid.hash, packed_git)) { + }else if (find_sha1_pack (oid.hash, the_repository->objects.packed_git)) {
Long line. (I'll refrain from commenting about any further ones.)
[...]
cache.h line 5
Quoted 7 lines
cache.h +2
-0+++ b/http-push.c@@ -1,4 +1,5 @@#include "cache.h" +#include "object-store.h" #include "commit.h" #include "tag.h" #include "blob.h"
Stray change?
http-walker.c line 10
Quoted 12 lines
http-walker.c +1
-0@@ -4,6 +4,7 @@#include "http.h" #include "list.h" #include "transport.h" +#include "object-store.h" #include "packfile.h" struct alt_base {
Same question.
http.c line 7
Quoted 12 lines
http.c +1
-0@@ -1,6 +1,7 @@#include "git-compat-util.h" #include "http.h" #include "config.h" +#include "object-store.h" #include "pack.h" #include "sideband.h" #include "run-command.h"
Likewise.
Quoted 36 lines
object-store.h +24
-0@@ -52,6 +52,30 @@ void add_to_alternates_memory(const char *dir);*/ struct strbuf *alt_scratch_buf (struct alternate_object_database *alt); +struct packed_git { +struct packed_git *next; +struct list_head mru; +struct pack_window *windows; +off_t pack_size; +const void *index_data; +size_t index_size; +uint32_t num_objects; +uint32_t num_bad_objects; +unsigned char *bad_object_sha1; +int index_version; +time_t mtime; +int pack_fd; +unsigned pack_local:1 , + pack_keep:1 , + freshened:1 , + do_not_close:1 , + pack_promisor:1 ; +unsigned char sha1[20 ]; +struct revindex_entry *revindex; +/* something like ".git/objects/pack/xxxxx.pack" */ +char pack_name[FLEX_ARRAY ];/* more */ +}; +struct raw_object_store {/* * Path to the repository's object store.
object-store.h struct raw_object_store { · line 91
Quoted 9 lines
object-store.h +6
-0@@ -59,10 +83,25 @@ struct raw_object_store {*/ char *objectdir; +struct packed_git *packed_git; +/* +* A most-recently-used ordered version of the packed_git list, which can +* be iterated instead of packed_git (and marked via mru_mark). +*/ +struct list_head packed_git_mru;
I don't understand the new part of the comment. Can you explain here, for me?
Is this meant as a list of related functions, an explanation of what the field is for, or something else?
object-store.h struct raw_object_store { · line 102
Quoted 12 lines
object-store.h +8
-1+struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; }; -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL } + +/* +* The mru list_head is supposed to be initialized using +* the LIST_HEAD macro, assigning prev/next to itself. +* However this doesn't work in this case as some compilers dislike +* that macro on member variables. Use NULL instead as that is defined +* and accepted, deferring the real init to prepare_packed_git_mru(). */
style nit: '*/' should be on its own line.
More importantly, we can avoid such an issue as described by Junio. :)
object-store.h struct raw_object_store { · line 103
Quoted 1 line
object-store.h +1
-0+#define __MRU_INIT { NULL, NULL }
Identifiers with leading underscores like this are in a reserved namespace for the language implementation --- we can't count on them being available for our own use.
Quoted 1 line
object-store.h +1
-0+#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
[...]
object.c void raw_object_store_clear(struct raw_object_store *o) · line 475
Quoted 13 lines
object.c +7
-0@@ -466,4 +466,11 @@ void raw_object_store_clear(struct raw_object_store *o)free_alt_odbs (o); o->alt_odb_tail =NULL ; + +while (!list_empty (&o->packed_git_mru)) +list_del (&o->packed_git_mru); +/* +* TODO: call close_all_packs once migrated to +* take an object store argument +*/
Can you say more about this TODO? Does this mean the patches are out of order (i.e. that raw_object_store_clear leaves behind a leak until a later patch)?
[...]
pack-check.c line 6
Quoted 9 lines
pack-check.c +1
-0@@ -1,5 +1,6 @@#include "cache.h" #include "pack.h" +#include "object-store.h" #include "pack-revindex.h" #include "progress.h" #include "packfile.h"
Another unexplained #include (I'll refrain from pointing out later ones).
The rest looks good.
Thanks,
Jonathan
Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store
Quoted 10 lines
Quoted 7 lines
2. Applying the semantic patch
contrib/coccinelle/refactoring/packed_git.coccito adjust callers. This semantic patch is placed in a sub directory of the coccinelle contrib dir, as this semantic patch is not expected to be of general usefulness; it is only useful during developing this series and merging it with other topics in flight. At a later date, just delete that semantic patch.Can the semantic patch go in the commit message instead? It is very brief.
done
Quoted 7 lines
Actually, I don't see this semantic patch in the diffstat. Is the commit message stale?
Quoted 3 lines
3. Applying line wrapping fixes from "make style" to break the resulting long lines.
Quoted 6 lines
Quoted 2 lines
4. Adding missing #includes of
repository.handobject-store.hwhere needed.Is there a way to automate this step? (I'm asking for my own reference when writing future patches, not because of any concern about the correctness of this one.)
no, for several reasons.
(a) I don't know how to write it in coccinelle, because
(b) I have not figured out the order in which we include headers, apart from "cache.h goes first", the rest of the includes sometimes looks like a random order, because different patches add new includes at different places. I have the impression, that (1) some add the include after all other existing includes or (2) try to figure out where to add the include to make sense in the existing include order or (3) sort alphabetically or (4) put it randomly, so the chance for a merge conflict with other series in flight decreases.
(c) I did it in a semi automated fashion:
while make fails:
add another include
The mess with including repository or object-store comes from the fact that I had v2 based on object-store, not the repository and cherry-picked this patch over to v3. Fixed all of the includes now.
Quoted 4 lines
Quoted 4 lines
object-store.h +0
-0@@ -59,10 +83,25 @@ struct raw_object_store {*/ char *objectdir;
Quoted 9 lines
object-store.h struct raw_object_store { · line 91
Quoted 6 lines
object-store.h +6
-0+struct packed_git *packed_git; +/* +* A most-recently-used ordered version of the packed_git list, which can +* be iterated instead of packed_git (and marked via mru_mark). +*/ +struct list_head packed_git_mru;I don't understand the new part of the comment. Can you explain here, for me?
cherrypicking error, fixed.
Quoted 11 lines
object-store.h struct raw_object_store { · line 102
Quoted 8 lines
object-store.h +7
-1-#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL } + +/* +* The mru list_head is supposed to be initialized using +* the LIST_HEAD macro, assigning prev/next to itself. +* However this doesn't work in this case as some compilers dislike +* that macro on member variables. Use NULL instead as that is defined +* and accepted, deferring the real init to prepare_packed_git_mru(). */ style nit: '*/' should be on its own line.
Quoted 1 line
More importantly, we can avoid such an issue as described by Junio. :)
See reply to Junio, I am not quite sure I like that.
PATCH v3 6/27 object-store: close all packs upon clearing the object store
In reply to Stefan Beller--- builtin/am.c | 2 +- builtin/clone.c | 2 +- builtin/fetch.c | 2 +- builtin/merge.c | 2 +- builtin/receive-pack.c | 2 +- object.c | 6 ++---- packfile.c | 4 ++-- packfile.h | 2 +- 8 files changed, 10 insertions(+), 12 deletions(-)
builtin/am.c +1 -1
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)*/ if (!state->rebasing) {am_destroy (state); -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }
builtin/clone.c +1 -1
@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)transport_disconnect (transport);if (option_dissociate) { -close_all_packs (); +close_all_packs (&the_repository->objects);dissociate_from_references (); }
builtin/fetch.c +1 -1
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)string_list_clear (&list,0 ); -close_all_packs (); +close_all_packs (&the_repository->objects);argv_array_pushl (&argv_gc_auto,"gc" ,"--auto" ,NULL );if (verbosity <0 )
builtin/merge.c +1 -1
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,* We ignore errors in 'gc --auto', since the * user should see them. */ -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }
builtin/receive-pack.c +1 -1
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix) proc.git_cmd =1 ; proc.argv = argv_gc_auto; -close_all_packs (); +close_all_packs (&the_repository->objects);if (!start_command (&proc)) {if (use_sideband)copy_to_sideband (proc.err,-1 ,NULL );
object.c +2 -4
@@ -4,6 +4,7 @@#include "tree.h" #include "commit.h" #include "tag.h" +#include "packfile.h" static struct object **obj_hash;static int nr_objs, obj_hash_size;
@@ -469,8 +470,5 @@ void raw_object_store_clear(struct raw_object_store *o)while (!list_empty (&o->packed_git_mru))list_del (&o->packed_git_mru); -/* -* TODO: call close_all_packs once migrated to -* take an object store argument -*/ +close_all_packs (o); }
packfile.c +2 -2
@@ -311,11 +311,11 @@ static void close_pack(struct packed_git *p)close_pack_index (p); } -void close_all_packs (void ) +void close_all_packs (struct raw_object_store *o) {struct packed_git *p; -for (p = the_repository->objects.packed_git; p; p = p->next) +for (p = o->packed_git; p; p = p->next)if (p->do_not_close)die ("BUG: want to close pack marked 'do-not-close'" );else
packfile.h +1 -1
@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void close_pack_windows (struct packed_git *); -extern void close_all_packs (void ); +extern void close_all_packs (struct raw_object_store *o);extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 06/27] object-store: close all packs upon clearing the object store
Quoted 1 line · Stefan Beller
Signed-off-by Stefan Beller <[email protected]>
Straight forward change, looks good.
Quoted 132 lines
--- builtin/am.c | 2 +- builtin/clone.c | 2 +- builtin/fetch.c | 2 +- builtin/merge.c | 2 +- builtin/receive-pack.c | 2 +- object.c | 6 ++---- packfile.c | 4 ++-- packfile.h | 2 +- 8 files changed, 10 insertions(+), 12 deletions(-)builtin/am.c +1
-1@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)*/ if (!state->rebasing) {am_destroy (state); -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }builtin/clone.c +1
-1@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)transport_disconnect (transport);if (option_dissociate) { -close_all_packs (); +close_all_packs (&the_repository->objects);dissociate_from_references (); }builtin/fetch.c +1
-1@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)string_list_clear (&list,0 ); -close_all_packs (); +close_all_packs (&the_repository->objects);argv_array_pushl (&argv_gc_auto,"gc" ,"--auto" ,NULL );if (verbosity <0 )builtin/merge.c +1
-1@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,* We ignore errors in 'gc --auto', since the * user should see them. */ -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }builtin/receive-pack.c +1
-1@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix) proc.git_cmd =1 ; proc.argv = argv_gc_auto; -close_all_packs (); +close_all_packs (&the_repository->objects);if (!start_command (&proc)) {if (use_sideband)copy_to_sideband (proc.err,-1 ,NULL );object.c +2
-4@@ -4,6 +4,7 @@#include "tree.h" #include "commit.h" #include "tag.h" +#include "packfile.h" static struct object **obj_hash;static int nr_objs, obj_hash_size;@@ -469,8 +470,5 @@ void raw_object_store_clear(struct raw_object_store *o)while (!list_empty (&o->packed_git_mru))list_del (&o->packed_git_mru); -/* -* TODO: call close_all_packs once migrated to -* take an object store argument -*/ +close_all_packs (o); }packfile.c +2
-2@@ -311,11 +311,11 @@ static void close_pack(struct packed_git *p)close_pack_index (p); } -void close_all_packs (void ) +void close_all_packs (struct raw_object_store *o) {struct packed_git *p; -for (p = the_repository->objects.packed_git; p; p = p->next) +for (p = o->packed_git; p; p = p->next)if (p->do_not_close)die ("BUG: want to close pack marked 'do-not-close'" );else packfile.h +1
-1@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void close_pack_windows (struct packed_git *); -extern void close_all_packs (void ); +extern void close_all_packs (struct raw_object_store *o);extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);Signature · 1 line
2.16.1.291.g4437f3f132-goog
Signature · 1 line
Brandon Williams
PATCH v3 7/27 pack: move prepare_packed_git_run_once to object store
In reply to Stefan BellerEach repository's object store can be initialized independently, so they must not share a run_once variable.
--- object-store.h | 8 +++++++- packfile.c | 7 +++---- 2 files changed, 10 insertions(+), 5 deletions(-)
object-store.h +7 -1
@@ -92,6 +92,12 @@ struct raw_object_store {struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; + +/* +* Whether packed_git has already been populated with this repository's +* packs. +*/ + unsigned packed_git_initialized : 1; };/*
@@ -101,7 +107,7 @@ struct raw_object_store {* that macro on member variables. Use NULL instead as that is defined * and accepted, deferring the real init to prepare_packed_git_mru(). */ #define __MRU_INIT { NULL, NULL } -#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +3 -4
@@ -884,12 +884,11 @@ static void prepare_packed_git_mru(void)list_add_tail (&p->mru, &the_repository->objects.packed_git_mru); } -static int prepare_packed_git_run_once =0 ;void prepare_packed_git (void ) {struct alternate_object_database *alt; -if (prepare_packed_git_run_once) +if (the_repository->objects.packed_git_initialized)return ;prepare_packed_git_one (get_object_directory (),1 );prepare_alt_odb ();
@@ -897,13 +896,13 @@ void prepare_packed_git(void)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();prepare_packed_git_mru (); - prepare_packed_git_run_once =1 ; + the_repository->objects.packed_git_initialized =1 ; }void reprepare_packed_git (void ) { approximate_object_count_valid =0 ; - prepare_packed_git_run_once =0 ; + the_repository->objects.packed_git_initialized =0 ;prepare_packed_git (); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 07/27] pack: move prepare_packed_git_run_once to object store
Quoted 5 lines · On Tue, 20 Feb 2018 17:54:10 -0800 Stefan Beller <[email protected]> wrote:
Each repository's object store can be initialized independently, so they must not share a
run_oncevariable.Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>
object-store.h struct raw_object_store { · line 110
Quoted 2 lines
object-store.h +1
-1-#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }
Optional: Trailing zeros are not needed in struct initializers.
PATCH v3 8/27 pack: move approximate object count to object store
In reply to Stefan BellerThe approximate_object_count() function maintains a rough count of objects in a repository to estimate how long object name abbreviates should be. Object names are scoped to a repository and the appropriate length may differ by repository, so the object count should not be global.
--- object-store.h | 10 +++++++++- packfile.c | 11 +++++------ 2 files changed, 14 insertions(+), 7 deletions(-)
object-store.h +9 -1
@@ -93,6 +93,14 @@ struct raw_object_store {struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; +/* +* A fast, rough count of the number of objects in the repository. +* These two fields are not meant for direct access. Use +* approximate_object_count() instead. +*/ +unsigned long approximate_object_count; + unsigned approximate_object_count_valid : 1; +/* * Whether packed_git has already been populated with this repository's * packs.
@@ -107,7 +115,7 @@ struct raw_object_store {* that macro on member variables. Use NULL instead as that is defined * and accepted, deferring the real init to prepare_packed_git_mru(). */ #define __MRU_INIT { NULL, NULL } -#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +5 -6
@@ -803,8 +803,6 @@ static void prepare_packed_git_one(char *objdir, int local)strbuf_release (&path); } -static int approximate_object_count_valid; -/* * Give a fast, rough count of the number of objects in the repository. This * ignores loose objects completely. If you have a lot of them, then either
@@ -814,8 +812,8 @@ static int approximate_object_count_valid;*/ unsigned long approximate_object_count (void ) { -static unsigned long count; -if (!approximate_object_count_valid) { +if (!the_repository->objects.approximate_object_count_valid) { +unsigned long count;struct packed_git *p;prepare_packed_git ();
@@ -825,8 +823,9 @@ unsigned long approximate_object_count(void)continue ; count += p->num_objects; } + the_repository->objects.approximate_object_count = count; } -return count; +return the_repository->objects.approximate_object_count; }static void *get_next_packed_git (const void *p)
@@ -901,7 +900,7 @@ void prepare_packed_git(void)void reprepare_packed_git (void ) { - approximate_object_count_valid =0 ; + the_repository->objects.approximate_object_count_valid =0 ; the_repository->objects.packed_git_initialized =0 ;prepare_packed_git (); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 08/27] pack: move approximate object count to object store
Quoted 18 lines · Stefan Beller
The
approximate_object_count()function maintains a rough count of objects in a repository to estimate how long object name abbreviates should be. Object names are scoped to a repository and the appropriate length may differ by repository, so the object count should not be global.Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>--- object-store.h | 10 +++++++++- packfile.c | 11 +++++------ 2 files changed, 14 insertions(+), 7 deletions(-)object-store.h +0
-0
object-store.h struct raw_object_store { · line 102
Quoted 10 lines
object-store.h +7
-0@@ -93,6 +93,14 @@ struct raw_object_store {struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; +/* +* A fast, rough count of the number of objects in the repository. +* These two fields are not meant for direct access. Use +* approximate_object_count() instead. +*/ +unsigned long approximate_object_count; + unsigned approximate_object_count_valid : 1;
Patch looks fine and is effectively a no-op, though what is the need for both of these variables? Maybe it can be simplified down to just use one? Just musing as its out of the scope of this patch and we probably shouldn't try to change that in this series.
Quoted 60 lines
object-store.h +2
-1+/* * Whether packed_git has already been populated with this repository's * packs. @@ -107,7 +115,7 @@ struct raw_object_store {* that macro on member variables. Use NULL instead as that is defined * and accepted, deferring the real init to prepare_packed_git_mru(). */ #define __MRU_INIT { NULL, NULL } -#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);packfile.c +5
-6@@ -803,8 +803,6 @@ static void prepare_packed_git_one(char *objdir, int local)strbuf_release (&path); } -static int approximate_object_count_valid; -/* * Give a fast, rough count of the number of objects in the repository. This * ignores loose objects completely. If you have a lot of them, then either @@ -814,8 +812,8 @@ static int approximate_object_count_valid;*/ unsigned long approximate_object_count (void ) { -static unsigned long count; -if (!approximate_object_count_valid) { +if (!the_repository->objects.approximate_object_count_valid) { +unsigned long count;struct packed_git *p;prepare_packed_git ();@@ -825,8 +823,9 @@ unsigned long approximate_object_count(void)continue ; count += p->num_objects; } + the_repository->objects.approximate_object_count = count; } -return count; +return the_repository->objects.approximate_object_count; }static void *get_next_packed_git (const void *p)@@ -901,7 +900,7 @@ void prepare_packed_git(void)void reprepare_packed_git (void ) { - approximate_object_count_valid =0 ; + the_repository->objects.approximate_object_count_valid =0 ; the_repository->objects.packed_git_initialized =0 ;prepare_packed_git (); }Signature · 1 line
2.16.1.291.g4437f3f132-goog
Signature · 1 line
Brandon Williams
Re: [PATCH 08/27] pack: move approximate object count to object store
Quoted 22 lines · Brandon Williams
Quoted 21 lines · Stefan Beller
The
approximate_object_count()function maintains a rough count of objects in a repository to estimate how long object name abbreviates should be. Object names are scoped to a repository and the appropriate length may differ by repository, so the object count should not be global.Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>--- object-store.h | 10 +++++++++- packfile.c | 11 +++++------ 2 files changed, 14 insertions(+), 7 deletions(-)object-store.h +0
-0@@ -93,6 +93,14 @@ struct raw_object_store {struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail;
Quoted 12 lines
object-store.h struct raw_object_store { · line 102
Quoted 7 lines
object-store.h +7
-0+/* +* A fast, rough count of the number of objects in the repository. +* These two fields are not meant for direct access. Use +* approximate_object_count() instead. +*/ +unsigned long approximate_object_count; + unsigned approximate_object_count_valid : 1;Patch looks fine and is effectively a no-op, though what is the need for both of these variables? Maybe it can be simplified down to just use one? Just musing as its out of the scope of this patch and we probably shouldn't try to change that in this series.
I agree we should. It was introduced in e323de4ad7f (sha1_file:
allow sha1_loose_object_info to handle arbitrary repositories, 2017-08-30) and I think it was seen as a clever optimization trick back then?
Re: [PATCH 08/27] pack: move approximate object count to object store
Quoted 8 lines · Stefan Beller
Quoted 8 lines
Quoted 7 lines
object-store.h +7
-0+/* +* A fast, rough count of the number of objects in the repository. +* These two fields are not meant for direct access. Use +* approximate_object_count() instead. +*/ +unsigned long approximate_object_count; + unsigned approximate_object_count_valid : 1;
Quoted 8 lines
Quoted 4 lines
Patch looks fine and is effectively a no-op, though what is the need for both of these variables? Maybe it can be simplified down to just use one? Just musing as its out of the scope of this patch and we probably shouldn't try to change that in this series.
I agree we should. It was introduced in
e323de4ad7f(sha1_file:
allowsha1_loose_object_infoto handle arbitrary repositories, 2017-08-30) and I think it was seen as a clever optimization trick back then?
I think you meant 8e3f52d778 (find_unique_abbrev: move logic out of get_short_sha1(), 2016-10-03)?
Yes, it was just to avoid the dual-meaning of "0" for "not set" and a repository with no packfiles. It would probably be fine to get rid of it. If you have no packfiles then you probably don't have enough objects to worry about micro-optimizing. And anyway, the "wasted" case wouldn't even make any syscalls (it would do a noop prepare_packed_git() and then realize the packed_git list is empty).
-Peff
Re: [PATCH 08/27] pack: move approximate object count to object store
Quoted 15 lines · Jeff King
Quoted 13 lines · Stefan Beller
Quoted 12 lines
object-store.h struct raw_object_store { · line 102
Quoted 7 lines
object-store.h +7
-0+/* +* A fast, rough count of the number of objects in the repository. +* These two fields are not meant for direct access. Use +* approximate_object_count() instead. +*/ +unsigned long approximate_object_count; + unsigned approximate_object_count_valid : 1;Patch looks fine and is effectively a no-op, though what is the need for both of these variables? Maybe it can be simplified down to just use one? Just musing as its out of the scope of this patch and we probably shouldn't try to change that in this series.
Quoted 6 lines
Quoted 3 lines
I agree we should. It was introduced in
e323de4ad7f(sha1_file:
allowsha1_loose_object_infoto handle arbitrary repositories, 2017-08-30) and I think it was seen as a clever optimization trick back then?I think you meant
8e3f52d778(find_unique_abbrev: move logic out ofget_short_sha1(), 2016-10-03)?
Yes, copy paste error.
Quoted 7 lines
Yes, it was just to avoid the dual-meaning of "0" for "not set" and a repository with no packfiles. It would probably be fine to get rid of it. If you have no packfiles then you probably don't have enough objects to worry about micro-optimizing. And anyway, the "wasted" case wouldn't even make any syscalls (it would do a noop
prepare_packed_git()and then realize thepacked_gitlist is empty).
Good point!
Quoted 2 lines
-Peff
PATCH v3 9/27 sha1_file: add raw_object_store argument to alt_odb_usable
In reply to Stefan BellerAdd a raw_object_store to alt_odb_usable to be more specific about which repository to act on. The choice of the repository is delegated to its only caller link_alt_odb_entry.
--- sha1_file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
sha1_file.c +5 -3
@@ -346,7 +346,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,/* * Return non-zero iff the path is usable as an alternate object database. */ -static int alt_odb_usable (struct strbuf *path,const char *normalized_objdir) +static int alt_odb_usable (struct raw_object_store *o, +struct strbuf *path, +const char *normalized_objdir) {struct alternate_object_database *alt;
@@ -362,7 +364,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)* Prevent the common mistake of listing the same * thing twice, or object directory itself. */ -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +for (alt = o->alt_odb_list; alt; alt = alt->next) {if (!fspathcmp (path->buf, alt->path))return 0 ; }
@@ -414,7 +416,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,while (pathbuf.len && pathbuf.buf[pathbuf.len -1 ] =='/' )strbuf_setlen (&pathbuf, pathbuf.len -1 ); -if (!alt_odb_usable (&pathbuf, normalized_objdir)) { +if (!alt_odb_usable (&the_repository->objects, &pathbuf, normalized_objdir)) {strbuf_release (&pathbuf);return -1 ; }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 09/27] sha1_file: add raw_object_store argument to alt_odb_usable
Quoted 5 lines · On Tue, 20 Feb 2018 17:54:12 -0800 Stefan Beller <[email protected]> wrote:
Add a
raw_object_storetoalt_odb_usableto be more specific about which repository to act on. The choice of the repository is delegated to its only callerlink_alt_odb_entry.Signed-off-by Stefan Beller <[email protected]>
I checked that alt_odb_usable no longer depends on any repository-like globals.
PATCH v3 10/27 sha1_file: add repository argument to link_alt_odb_entry
In reply to Stefan BellerAdd a repository argument to allow the link_alt_odb_entry caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
Since the implementation does not yet work with other repositories, use a wrapper macro to enforce that the caller passes in the_repository as the first argument. It would be more appealing to use BUILD_ASSERT_OR_ZERO to enforce this, but that doesn't work because it requires a compile-time constant and common compilers like gcc 4.8.4 do not consider "r == the_repository" a compile-time constant.
This and the following three patches add repository arguments to link_alt_odb_entry, read_info_alternates, link_alt_odb_entries and prepare_alt_odb. Three out of the four functions are found in a recursive call chain, calling each other, and one of them accesses the repositories objectdir (which was migrated; it was an obvious choice) and ignore_env (which we need to keep in the repository struct for clarify); hence we will pass through the repository unlike just the object store object + the ignore_env flag.
--- sha1_file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
sha1_file.c +5 -3
@@ -390,8 +390,9 @@ static int alt_odb_usable(struct raw_object_store *o,* terminating NUL. */ static void read_info_alternates (const char * relative_base,int depth); -static int link_alt_odb_entry (const char *entry,const char *relative_base, -int depth,const char *normalized_objdir) +#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n) +static int link_alt_odb_entry_the_repository (const char *entry, +const char *relative_base,int depth,const char *normalized_objdir) {struct alternate_object_database *ent;struct strbuf pathbuf =STRBUF_INIT ;
@@ -488,7 +489,8 @@ static void link_alt_odb_entries(const char *alt, int sep, alt =parse_alt_odb_entry (alt, sep, &entry);if (!entry.len)continue ; -link_alt_odb_entry (entry.buf, relative_base, depth, objdirbuf.buf); +link_alt_odb_entry (the_repository, entry.buf, + relative_base, depth, objdirbuf.buf); }strbuf_release (&entry);strbuf_release (&objdirbuf);
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v3 11/27 sha1_file: add repository argument to read_info_alternates
In reply to Stefan BellerSee previous patch for explanation.
--- sha1_file.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
sha1_file.c +7 -4
@@ -389,7 +389,9 @@ static int alt_odb_usable(struct raw_object_store *o,* SHA1, an extra slash for the first level indirection, and the * terminating NUL. */ -static void read_info_alternates (const char * relative_base,int depth); +#define read_info_alternates (r, rb, d) read_info_alternates_##r(rb, d) +static void read_info_alternates_the_repository (const char *relative_base, +int depth);#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)static int link_alt_odb_entry_the_repository (const char *entry,const char *relative_base,int depth,const char *normalized_objdir)
@@ -430,7 +432,7 @@ static int link_alt_odb_entry_the_repository(const char *entry, ent->next =NULL ;/* recursively add alternates */ -read_info_alternates (pathbuf.buf, depth +1 ); +read_info_alternates (the_repository, pathbuf.buf, depth +1 );strbuf_release (&pathbuf);return 0 ;
@@ -496,7 +498,8 @@ static void link_alt_odb_entries(const char *alt, int sep,strbuf_release (&objdirbuf); } -static void read_info_alternates (const char * relative_base,int depth) +static void read_info_alternates_the_repository (const char *relative_base, +int depth) {char *path;struct strbuf buf =STRBUF_INIT ;
@@ -680,7 +683,7 @@ void prepare_alt_odb(void) &the_repository->objects.alt_odb_list;link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 ); -read_info_alternates (get_object_directory (),0 ); +read_info_alternates (the_repository,get_object_directory (),0 ); }/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v3 12/27 sha1_file: add repository argument to link_alt_odb_entries
In reply to Stefan BellerSee previous patch for explanation.
--- sha1_file.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
sha1_file.c +13 -6
@@ -467,8 +467,12 @@ static const char *parse_alt_odb_entry(const char *string,return end; } -static void link_alt_odb_entries (const char *alt,int sep, -const char *relative_base,int depth) +#define link_alt_odb_entries (r, a, s, rb, d) \ + link_alt_odb_entries_##r(a, s, rb, d) +static void link_alt_odb_entries_the_repository (const char *alt, +int sep, +const char *relative_base, +int depth) {struct strbuf objdirbuf =STRBUF_INIT ;struct strbuf entry =STRBUF_INIT ;
@@ -511,7 +515,7 @@ static void read_info_alternates_the_repository(const char *relative_base,return ; } -link_alt_odb_entries (buf.buf,'\n' , relative_base, depth); +link_alt_odb_entries (the_repository, buf.buf,'\n' , relative_base, depth);strbuf_release (&buf);free (path); }
@@ -565,7 +569,8 @@ void add_to_alternates_file(const char *reference)if (commit_lock_file (&lock))die_errno ("unable to move new alternates file into place" );if (the_repository->objects.alt_odb_tail) -link_alt_odb_entries (reference,'\n' ,NULL ,0 ); +link_alt_odb_entries (the_repository, reference, +'\n' ,NULL ,0 ); }free (alts); }
@@ -578,7 +583,8 @@ void add_to_alternates_memory(const char *reference)*/ prepare_alt_odb (); -link_alt_odb_entries (reference,'\n' ,NULL ,0 ); +link_alt_odb_entries (the_repository, reference, +'\n' ,NULL ,0 ); }/*
@@ -681,7 +687,8 @@ void prepare_alt_odb(void) the_repository->objects.alt_odb_tail = &the_repository->objects.alt_odb_list; -link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (the_repository, alt, +PATH_SEP ,NULL ,0 );read_info_alternates (the_repository,get_object_directory (),0 ); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v3 13/27 sha1_file: add repository argument to prepare_alt_odb
In reply to Stefan BellerSee previous patch for explanation.
While at it, move the declaration to object-store.h, where it should be easier to find.
--- builtin/fsck.c | 2 +- object-store.h | 3 ++- packfile.c | 2 +- sha1_file.c | 13 +++++++------ sha1_name.c | 3 ++- 5 files changed, 13 insertions(+), 10 deletions(-)
builtin/fsck.c +1 -1
@@ -717,7 +717,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) }else {fsck_object_dir (get_object_directory ()); -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)fsck_object_dir (alt->path);
object-store.h +2 -1
@@ -20,7 +20,8 @@ struct alternate_object_database {char path[FLEX_ARRAY ]; }; -void prepare_alt_odb (void ); +#define prepare_alt_odb (r) prepare_alt_odb_##r() +void prepare_alt_odb_the_repository (void );char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);int foreach_alt_odb (alt_odb_fn ,void *);
packfile.c +1 -1
@@ -890,7 +890,7 @@ void prepare_packed_git(void)if (the_repository->objects.packed_git_initialized)return ;prepare_packed_git_one (get_object_directory (),1 ); -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();
sha1_file.c +7 -6
@@ -23,6 +23,7 @@#include "sha1-lookup.h" #include "bulk-checkin.h" #include "repository.h" +#include "object-store.h" #include "streaming.h" #include "dir.h" #include "list.h"
@@ -581,7 +582,7 @@ void add_to_alternates_memory(const char *reference)* Make sure alternates are initialized, or else our entry may be * overwritten when they are. */ -prepare_alt_odb (); +prepare_alt_odb (the_repository);link_alt_odb_entries (the_repository, reference,'\n' ,NULL ,0 );
@@ -667,7 +668,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)struct alternate_object_database *ent;int r =0 ; -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) { r =fn (ent, cb);if (r)
@@ -676,7 +677,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb (void ) +void prepare_alt_odb_the_repository (void ) {const char *alt;
@@ -730,7 +731,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)static int check_and_freshen_nonlocal (const unsigned char *sha1,int freshen) {struct alternate_object_database *alt; -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {const char *path =alt_sha1_path (alt, sha1);if (check_and_freshen_file (path, freshen))
@@ -889,7 +890,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,if (!lstat (*path, st))return 0 ; -prepare_alt_odb (); +prepare_alt_odb (the_repository); errno =ENOENT ;for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);
@@ -920,7 +921,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)return fd; most_interesting_errno = errno; -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);
sha1_name.c +2 -1
@@ -11,6 +11,7 @@#include "sha1-array.h" #include "packfile.h" #include "repository.h" +#include "object-store.h" static int get_oid_oneline (const char *,struct object_id *,struct commit_list *);
@@ -353,7 +354,7 @@ static int init_object_disambiguation(const char *name, int len, ds->len = len; ds->hex_pfx[len] ='\0' ; -prepare_alt_odb (); +prepare_alt_odb (the_repository);return 0 ; }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb
Hi,
Quoted 4 lines · Stefan Beller
See previous patch for explanation.
While at it, move the declaration to
object-store.h, where it should be easier to find.
Which declaration? It looks like prepare_alt_odb is already in object-store.h.
[...]
builtin/fsck.c int cmd_fsck(int argc, const char **argv, const char *prefix) · line 720
Quoted 8 lines
builtin/fsck.c +1
-1@@ -717,7 +717,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) }else {fsck_object_dir (get_object_directory ()); -prepare_alt_odb (); +prepare_alt_odb (the_repository);
Patch 2 added a #include of "repository.h". Good.
(I checked because with the definition of prepare_alt_odb as a macro, the function call would compile correctly even if the_repository weren't in scope, but we want to include what we use as a matter of style/maintainability.)
[...]
packfile.c void prepare_packed_git(void) · line 893
Quoted 8 lines
packfile.c +1
-1@@ -890,7 +890,7 @@ void prepare_packed_git(void)if (the_repository->objects.packed_git_initialized)return ;prepare_packed_git_one (get_object_directory (),1 ); -prepare_alt_odb (); +prepare_alt_odb (the_repository);
Also good, since patch 3 added a #include of "repository.h".
[...]
sha1_file.c line 26
Quoted 7 lines
sha1_file.c +1
-0@@ -23,6 +23,7 @@#include "sha1-lookup.h" #include "bulk-checkin.h" #include "repository.h" +#include "object-store.h"
Should this #include have been added in an earlier patch, since the file both uses and defines prepare_alt_odb, which is declared there?
The rest looks good.
Thanks,
Jonathan
Re: [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb
Quoted 10 lines
sha1_file.c line 26
Quoted 7 lines
sha1_file.c +1
-0@@ -23,6 +23,7 @@#include "sha1-lookup.h" #include "bulk-checkin.h" #include "repository.h" +#include "object-store.h" Should this #include have been added in an earlier patch, since the file both uses and defines
prepare_alt_odb, which is declared there?
This is another superflous include, dropped it. Thanks for catching.
PATCH v3 14/27 sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
In reply to Stefan BellerActually this also allows read_info_alternates and link_alt_odb_entry to handle arbitrary repositories, but link_alt_odb_entries is the most interesting function in this set of functions, hence the commit subject.
These functions span a strongly connected component in the function graph, i.e. the recursive call chain might look like
-> link_alt_odb_entries
-> link_alt_odb_entry
-> read_info_alternates
-> link_alt_odb_entriesThat is why we need to convert all these functions at the same time.
--- object-store.h | 4 ++++ sha1_file.c | 36 ++++++++++++++++-------------------- 2 files changed, 20 insertions(+), 20 deletions(-)
object-store.h +4 -0
@@ -18,6 +18,10 @@ struct alternate_object_database {char loose_objects_subdir_seen[256 ];struct oid_array loose_objects_cache; +/* +* Path to the alternative object store. If this is a relative path, +* it is relative to the current working directory. +*/ char path[FLEX_ARRAY ]; };#define prepare_alt_odb (r) prepare_alt_odb_##r()
sha1_file.c +16 -20
@@ -390,11 +390,10 @@ static int alt_odb_usable(struct raw_object_store *o,* SHA1, an extra slash for the first level indirection, and the * terminating NUL. */ -#define read_info_alternates (r, rb, d) read_info_alternates_##r(rb, d) -static void read_info_alternates_the_repository (const char *relative_base, -int depth); -#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n) -static int link_alt_odb_entry_the_repository (const char *entry, +static void read_info_alternates (struct repository *r, +const char *relative_base, +int depth); +static int link_alt_odb_entry (struct repository *r,const char *entry,const char *relative_base,int depth,const char *normalized_objdir) {struct alternate_object_database *ent;
@@ -420,7 +419,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,while (pathbuf.len && pathbuf.buf[pathbuf.len -1 ] =='/' )strbuf_setlen (&pathbuf, pathbuf.len -1 ); -if (!alt_odb_usable (&the_repository->objects, &pathbuf, normalized_objdir)) { +if (!alt_odb_usable (&r->objects, &pathbuf, normalized_objdir)) {strbuf_release (&pathbuf);return -1 ; }
@@ -428,12 +427,12 @@ static int link_alt_odb_entry_the_repository(const char *entry, ent =alloc_alt_odb (pathbuf.buf);/* add the alternate entry */ - *the_repository->objects.alt_odb_tail = ent; - the_repository->objects.alt_odb_tail = &(ent->next); + *r->objects.alt_odb_tail = ent; + r->objects.alt_odb_tail = &(ent->next); ent->next =NULL ;/* recursively add alternates */ -read_info_alternates (the_repository, pathbuf.buf, depth +1 ); +read_info_alternates (r, pathbuf.buf, depth +1 );strbuf_release (&pathbuf);return 0 ;
@@ -468,12 +467,8 @@ static const char *parse_alt_odb_entry(const char *string,return end; } -#define link_alt_odb_entries (r, a, s, rb, d) \ - link_alt_odb_entries_##r(a, s, rb, d) -static void link_alt_odb_entries_the_repository (const char *alt, -int sep, -const char *relative_base, -int depth) +static void link_alt_odb_entries (struct repository *r,const char *alt, +int sep,const char *relative_base,int depth) {struct strbuf objdirbuf =STRBUF_INIT ;struct strbuf entry =STRBUF_INIT ;
@@ -487,7 +482,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,return ; } -strbuf_add_absolute_path (&objdirbuf,get_object_directory ()); +strbuf_add_absolute_path (&objdirbuf, r->objects.objectdir);if (strbuf_normalize_path (&objdirbuf) <0 )die ("unable to normalize object directory: %s" , objdirbuf.buf);
@@ -496,15 +491,16 @@ static void link_alt_odb_entries_the_repository(const char *alt, alt =parse_alt_odb_entry (alt, sep, &entry);if (!entry.len)continue ; -link_alt_odb_entry (the_repository, entry.buf, +link_alt_odb_entry (r, entry.buf, relative_base, depth, objdirbuf.buf); }strbuf_release (&entry);strbuf_release (&objdirbuf); } -static void read_info_alternates_the_repository (const char *relative_base, -int depth) +static void read_info_alternates (struct repository *r, +const char *relative_base, +int depth) {char *path;struct strbuf buf =STRBUF_INIT ;
@@ -516,7 +512,7 @@ static void read_info_alternates_the_repository(const char *relative_base,return ; } -link_alt_odb_entries (the_repository, buf.buf,'\n' , relative_base, depth); +link_alt_odb_entries (r, buf.buf,'\n' , relative_base, depth);strbuf_release (&buf);free (path); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
Quoted 12 lines · Stefan Beller
Actually this also allows
read_info_alternatesandlink_alt_odb_entryto handle arbitrary repositories, butlink_alt_odb_entriesis the most interesting function in this set of functions, hence the commit subject.These functions span a strongly connected component in the function graph, i.e. the recursive call chain might look like
->
link_alt_odb_entries-> link_alt_odb_entry -> read_info_alternates -> link_alt_odb_entries
Quoted 5 lines
That is why we need to convert all these functions at the same time.
Signed-off-by Jonathan Nieder <[email protected]>Signed-off-by Stefan Beller <[email protected]>---
Looks good.
Quoted 9 lines
object-store.h | 4 ++++ sha1_file.c | 36 ++++++++++++++++-------------------- 2 files changed, 20 insertions(+), 20 deletions(-)object-store.h +0
-0
object-store.h struct alternate_object_database { · line 24
Quoted 7 lines
object-store.h +4
-0@@ -18,6 +18,10 @@ struct alternate_object_database {char loose_objects_subdir_seen[256 ];struct oid_array loose_objects_cache; +/* +* Path to the alternative object store. If this is a relative path, +* it is relative to the current working directory. +*/
Thanks for adding the comment.
Quoted 104 lines
object-store.h +0
-0char path[FLEX_ARRAY ]; };#define prepare_alt_odb (r) prepare_alt_odb_##r()sha1_file.c +16
-20@@ -390,11 +390,10 @@ static int alt_odb_usable(struct raw_object_store *o,* SHA1, an extra slash for the first level indirection, and the * terminating NUL. */ -#define read_info_alternates (r, rb, d) read_info_alternates_##r(rb, d) -static void read_info_alternates_the_repository (const char *relative_base, -int depth); -#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n) -static int link_alt_odb_entry_the_repository (const char *entry, +static void read_info_alternates (struct repository *r, +const char *relative_base, +int depth); +static int link_alt_odb_entry (struct repository *r,const char *entry,const char *relative_base,int depth,const char *normalized_objdir) {struct alternate_object_database *ent;@@ -420,7 +419,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,while (pathbuf.len && pathbuf.buf[pathbuf.len -1 ] =='/' )strbuf_setlen (&pathbuf, pathbuf.len -1 ); -if (!alt_odb_usable (&the_repository->objects, &pathbuf, normalized_objdir)) { +if (!alt_odb_usable (&r->objects, &pathbuf, normalized_objdir)) {strbuf_release (&pathbuf);return -1 ; }@@ -428,12 +427,12 @@ static int link_alt_odb_entry_the_repository(const char *entry, ent =alloc_alt_odb (pathbuf.buf);/* add the alternate entry */ - *the_repository->objects.alt_odb_tail = ent; - the_repository->objects.alt_odb_tail = &(ent->next); + *r->objects.alt_odb_tail = ent; + r->objects.alt_odb_tail = &(ent->next); ent->next =NULL ;/* recursively add alternates */ -read_info_alternates (the_repository, pathbuf.buf, depth +1 ); +read_info_alternates (r, pathbuf.buf, depth +1 );strbuf_release (&pathbuf);return 0 ;@@ -468,12 +467,8 @@ static const char *parse_alt_odb_entry(const char *string,return end; } -#define link_alt_odb_entries (r, a, s, rb, d) \ - link_alt_odb_entries_##r(a, s, rb, d) -static void link_alt_odb_entries_the_repository (const char *alt, -int sep, -const char *relative_base, -int depth) +static void link_alt_odb_entries (struct repository *r,const char *alt, +int sep,const char *relative_base,int depth) {struct strbuf objdirbuf =STRBUF_INIT ;struct strbuf entry =STRBUF_INIT ;@@ -487,7 +482,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,return ; } -strbuf_add_absolute_path (&objdirbuf,get_object_directory ()); +strbuf_add_absolute_path (&objdirbuf, r->objects.objectdir);if (strbuf_normalize_path (&objdirbuf) <0 )die ("unable to normalize object directory: %s" , objdirbuf.buf);@@ -496,15 +491,16 @@ static void link_alt_odb_entries_the_repository(const char *alt, alt =parse_alt_odb_entry (alt, sep, &entry);if (!entry.len)continue ; -link_alt_odb_entry (the_repository, entry.buf, +link_alt_odb_entry (r, entry.buf, relative_base, depth, objdirbuf.buf); }strbuf_release (&entry);strbuf_release (&objdirbuf); } -static void read_info_alternates_the_repository (const char *relative_base, -int depth) +static void read_info_alternates (struct repository *r, +const char *relative_base, +int depth) {char *path;struct strbuf buf =STRBUF_INIT ;@@ -516,7 +512,7 @@ static void read_info_alternates_the_repository(const char *relative_base,return ; } -link_alt_odb_entries (the_repository, buf.buf,'\n' , relative_base, depth); +link_alt_odb_entries (r, buf.buf,'\n' , relative_base, depth);strbuf_release (&buf);free (path); }Signature · 1 line
2.16.1.291.g4437f3f132-goog
Signature · 1 line
Brandon Williams
PATCH v3 15/27 sha1_file: allow prepare_alt_odb to handle arbitrary repositories
In reply to Stefan Beller--- object-store.h | 3 +-- sha1_file.c | 21 +++++++++++---------- 2 files changed, 12 insertions(+), 12 deletions(-)
object-store.h +1 -2
@@ -24,8 +24,7 @@ struct alternate_object_database {*/ char path[FLEX_ARRAY ]; }; -#define prepare_alt_odb (r) prepare_alt_odb_##r() -void prepare_alt_odb_the_repository (void ); +void prepare_alt_odb (struct repository *r);char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);int foreach_alt_odb (alt_odb_fn ,void *);
sha1_file.c +11 -10
@@ -673,21 +673,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb_the_repository (void ) +void prepare_alt_odb (struct repository *r) { -const char *alt; - -if (the_repository->objects.alt_odb_tail) +if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); + r->objects.alt_odb_tail = &r->objects.alt_odb_list; + +if (!r->ignore_env) { +const char *alt =getenv (ALTERNATE_DB_ENVIRONMENT ); +if (!alt) + alt ="" ; - the_repository->objects.alt_odb_tail = - &the_repository->objects.alt_odb_list; -link_alt_odb_entries (the_repository, alt, -PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); + } -read_info_alternates (the_repository,get_object_directory (),0 ); +read_info_alternates (r, r->objects.objectdir,0 ); }/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
Quoted 6 lines · On Tue, 20 Feb 2018 17:54:18 -0800 Stefan Beller <[email protected]> wrote:
sha1_file.c +1
-4-void prepare_alt_odb_the_repository (void ) +void prepare_alt_odb (struct repository *r) { -const char *alt; - -if (the_repository->objects.alt_odb_tail)
sha1_file.c int foreach_alt_odb(alt_odb_fn fn, void *cb) · line 686
Quoted 10 lines
sha1_file.c +7
-1+if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); + r->objects.alt_odb_tail = &r->objects.alt_odb_list; + +if (!r->ignore_env) { +const char *alt =getenv (ALTERNATE_DB_ENVIRONMENT ); +if (!alt) + alt ="" ;
alt can be NULL, just like in the existing code, so these 2 lines are unnecessary. (link_alt_odb_entries() will just do nothing in either case.)
(I also think that the check of absence of alt should be done by the caller of link_alt_odb_entries(), not by link_alt_odb_entries() itself, but that is much beyond the scope of this patch set.)
Quoted 11 lines
sha1_file.c +3
-5- the_repository->objects.alt_odb_tail = - &the_repository->objects.alt_odb_list; -link_alt_odb_entries (the_repository, alt, -PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); + } -read_info_alternates (the_repository,get_object_directory (),0 ); +read_info_alternates (r, r->objects.objectdir,0 ); }
Re: [PATCH 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
Quoted 24 lines · Jonathan Tan
Quoted 6 lines · On Tue, 20 Feb 2018 17:54:18 -0800 Stefan Beller <[email protected]> wrote:
sha1_file.c +1
-4-void prepare_alt_odb_the_repository (void ) +void prepare_alt_odb (struct repository *r) { -const char *alt; - -if (the_repository->objects.alt_odb_tail)sha1_file.c int foreach_alt_odb(alt_odb_fn fn, void *cb) · line 686
Quoted 10 lines
sha1_file.c +7
-1+if (r->objects.alt_odb_tail)return ; - alt =getenv (ALTERNATE_DB_ENVIRONMENT ); + r->objects.alt_odb_tail = &r->objects.alt_odb_list; + +if (!r->ignore_env) { +const char *alt =getenv (ALTERNATE_DB_ENVIRONMENT ); +if (!alt) + alt ="" ;alt can be NULL, just like in the existing code, so these 2 lines are unnecessary. (
link_alt_odb_entries()will just do nothing in either case.)
Quoted 4 lines
(I also think that the check of absence of alt should be done by the caller of
link_alt_odb_entries(), not bylink_alt_odb_entries()itself, but that is much beyond the scope of this patch set.)
reverted to a literal translation of s/the_repository/r/, I forget why I implemented this change in the first place. dropped for now.
Thanks,
Stefan
PATCH v3 16/27 sha1_file: add repository argument to sha1_file_name
In reply to Stefan BellerAdd a repository argument to allow sha1_file_name callers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
While at it, move the declaration to object-store.h, where it should be easier to find.
--- cache.h | 6 ------ http-walker.c | 3 ++- http.c | 5 ++--- object-store.h | 7 +++++++ sha1_file.c | 10 +++++----- 5 files changed, 16 insertions(+), 15 deletions(-)
cache.h +0 -6
@@ -961,12 +961,6 @@ extern void check_repository_format(void);#define DATA_CHANGED 0x0020#define TYPE_CHANGED 0x0040 -/* -* Put in `buf` the name of the file in the local object database that -* would be used to store a loose object with the specified sha1. -*/ -extern void sha1_file_name (struct strbuf *buf,const unsigned char *sha1); -/* * Return an abbreviated sha1 unique within this repository's object database. * The result will be at least `len` characters long, and will be NUL
http-walker.c +2 -1
@@ -1,4 +1,5 @@#include "cache.h" +#include "repository.h" #include "commit.h" #include "walker.h" #include "http.h"
@@ -546,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1) ret =error ("File %s has bad hash" , hex); }else if (req->rename <0 ) {struct strbuf buf =STRBUF_INIT ; -sha1_file_name (&buf, req->sha1); +sha1_file_name (the_repository, &buf, req->sha1); ret =error ("unable to write sha1 filename %s" , buf.buf);strbuf_release (&buf); }
http.c +2 -3
@@ -2247,7 +2247,7 @@ struct http_object_request *new_http_object_request(const char *base_url,hashcpy (freq->sha1, sha1); freq->localfile =-1 ; -sha1_file_name (&filename, sha1); +sha1_file_name (the_repository, &filename, sha1);snprintf (freq->tmpfile,sizeof (freq->tmpfile),"%s.temp" , filename.buf);
@@ -2396,8 +2396,7 @@ int finish_http_object_request(struct http_object_request *freq)unlink_or_warn (freq->tmpfile);return -1 ; } - -sha1_file_name (&filename, freq->sha1); +sha1_file_name (the_repository, &filename, freq->sha1); freq->rename =finalize_object_file (freq->tmpfile, filename.buf);strbuf_release (&filename);
object-store.h +7 -0
@@ -123,4 +123,11 @@ struct raw_object_store {void raw_object_store_clear (struct raw_object_store *o); +/* +* Put in `buf` the name of the file in the local object database that +* would be used to store a loose object with the specified sha1. +*/ +#define sha1_file_name (r, b, s) sha1_file_name_##r(b, s) +void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1); +#endif /* OBJECT_STORE_H */
sha1_file.c +5 -5
@@ -323,7 +323,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1) } } -void sha1_file_name (struct strbuf *buf,const unsigned char *sha1) +void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1) {strbuf_addstr (buf,get_object_directory ());strbuf_addch (buf,'/' );
@@ -720,7 +720,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&buf, sha1); +sha1_file_name (the_repository, &buf, sha1);return check_and_freshen_file (buf.buf, freshen); }
@@ -881,7 +881,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&buf, sha1); +sha1_file_name (the_repository, &buf, sha1); *path = buf.buf;if (!lstat (*path, st))
@@ -910,7 +910,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&buf, sha1); +sha1_file_name (the_repository, &buf, sha1); *path = buf.buf; fd =git_open (*path);
@@ -1595,7 +1595,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,static struct strbuf filename =STRBUF_INIT ;strbuf_reset (&filename); -sha1_file_name (&filename, sha1); +sha1_file_name (the_repository, &filename, sha1); fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 16/27] sha1_file: add repository argument to sha1_file_name
Quoted 10 lines · Stefan Beller
Add a repository argument to allow
sha1_file_namecallers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other thanthe_repositoryyet.As with the previous commits, use a macro to catch callers passing a repository other than
the_repositoryat compile time.While at it, move the declaration to
object-store.h, where it should be easier to find.
Seems like we may want to make a sha1-file.h or an oid-file.h or something like that at some point as that seems like a better place for the function than in the object-store.h file?
Signature · 1 line
Brandon Williams
Re: [PATCH 16/27] sha1_file: add repository argument to sha1_file_name
Quoted 9 lines · Brandon Williams
Quoted 8 lines · Stefan Beller
Add a repository argument to allow
sha1_file_namecallers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other thanthe_repositoryyet.As with the previous commits, use a macro to catch callers passing a repository other than
the_repositoryat compile time.
Quoted 6 lines
Quoted 2 lines
While at it, move the declaration to
object-store.h, where it should be easier to find.Seems like we may want to make a
sha1-file.hor anoid-file.hor something like that at some point as that seems like a better place for the function than in theobject-store.hfile?
It depends what our long term goal is.
Do we want header and source file name to match for each function? Or do we want a coarser set of headers, such that we have a broad object-store.h, but that is implemented in sha1_file.c, packfile.c for the parts of the raw_objectstore and other .c files for the higher levels of the object store?
For now I'd just keep it in object-store.h as moving out just a couple functions seems less consistent?
Stefan
Re: [PATCH 16/27] sha1_file: add repository argument to sha1_file_name
Quoted 24 lines · Stefan Beller
Quoted 9 lines · Brandon Williams
Quoted 8 lines · Stefan Beller
Add a repository argument to allow
sha1_file_namecallers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other thanthe_repositoryyet.As with the previous commits, use a macro to catch callers passing a repository other than
the_repositoryat compile time.Quoted 6 lines
Quoted 2 lines
While at it, move the declaration to
object-store.h, where it should be easier to find.Seems like we may want to make a
sha1-file.hor anoid-file.hor something like that at some point as that seems like a better place for the function than in theobject-store.hfile?It depends what our long term goal is.
Do we want header and source file name to match for each function? Or do we want a coarser set of headers, such that we have a broadobject-store.h, but that is implemented insha1_file.c,packfile.cfor the parts of theraw_objectstoreand other .c files for the higher levels of the object store?
Quoted 2 lines
For now I'd just keep it in
object-store.has moving out just a couple functions seems less consistent?
Fair enough :)
Signature · 1 line
Brandon Williams
PATCH v3 17/27 sha1_file: add repository argument to stat_sha1_file
In reply to Stefan BellerAdd a repository argument to allow the stat_sha1_file caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
sha1_file.c +5 -4
@@ -874,8 +874,9 @@ int git_open_cloexec(const char *name, int flags)* Note that it may point to static storage and is only valid until another * call to sha1_file_name(), etc. */ -static int stat_sha1_file (const unsigned char *sha1,struct stat *st, -const char **path) +#define stat_sha1_file (r, s, st, p) stat_sha1_file_##r(s, st, p) +static int stat_sha1_file_the_repository (const unsigned char *sha1, +struct stat *st,const char **path) {struct alternate_object_database *alt;static struct strbuf buf =STRBUF_INIT ;
@@ -1181,7 +1182,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {const char *path;struct stat st; -if (stat_sha1_file (sha1, &st, &path) <0 ) +if (stat_sha1_file (the_repository, sha1, &st, &path) <0 )return -1 ;if (oi->disk_sizep) *oi->disk_sizep = st.st_size;
@@ -1395,7 +1396,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,die ("replacement %s not found for %s" ,sha1_to_hex (repl),sha1_to_hex (sha1)); -if (!stat_sha1_file (repl, &st, &path)) +if (!stat_sha1_file (the_repository, repl, &st, &path))die ("loose object %s (stored in %s) is corrupt" ,sha1_to_hex (repl), path);
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 17/27] sha1_file: add repository argument to stat_sha1_file
Hi,
Quoted 5 lines · Stefan Beller
Add a repository argument to allow the
stat_sha1_filecaller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other thanthe_repositoryyet.
Quoted 8 lines
As with the previous commits, use a macro to catch callers passing a repository other than
the_repositoryat compile time.Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>--- sha1_file.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
Simple and obviously correct.
PATCH v3 18/27 sha1_file: add repository argument to open_sha1_file
In reply to Stefan BellerAdd a repository argument to allow the open_sha1_file caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
sha1_file.c +4 -2
@@ -903,7 +903,9 @@ static int stat_sha1_file_the_repository(const unsigned char *sha1,* Like stat_sha1_file(), but actually open the object and return the * descriptor. See the caveats on the "path" parameter above. */ -static int open_sha1_file (const unsigned char *sha1,const char **path) +#define open_sha1_file (r, s, p) open_sha1_file_##r(s, p) +static int open_sha1_file_the_repository (const unsigned char *sha1, +const char **path) {int fd;struct alternate_object_database *alt;
@@ -946,7 +948,7 @@ static void *map_sha1_file_1(const char *path,if (path) fd =git_open (path);else - fd =open_sha1_file (sha1, &path); + fd =open_sha1_file (the_repository, sha1, &path); map =NULL ;if (fd >=0 ) {struct stat st;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v3 19/27 sha1_file: add repository argument to map_sha1_file_1
In reply to Stefan BellerAdd a repository argument to allow the map_sha1_file_1 caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
sha1_file.c +6 -5
@@ -938,9 +938,10 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,* Map the loose object at "path" if it is not NULL, or the path found by * searching for a loose object named "sha1". */ -static void *map_sha1_file_1 (const char *path, -const unsigned char *sha1, -unsigned long *size) +#define map_sha1_file_1 (r, p, s, si) map_sha1_file_1_##r(p, s, si) +static void *map_sha1_file_1_the_repository (const char *path, +const unsigned char *sha1, +unsigned long *size) {void *map;int fd;
@@ -969,7 +970,7 @@ static void *map_sha1_file_1(const char *path,void *map_sha1_file (const unsigned char *sha1,unsigned long *size) { -return map_sha1_file_1 (NULL , sha1, size); +return map_sha1_file_1 (the_repository,NULL , sha1, size); } static int unpack_sha1_short_header(git_zstream *stream,
@@ -2199,7 +2200,7 @@ int read_loose_object(const char *path, *contents =NULL ; - map =map_sha1_file_1 (path,NULL , &mapsize); + map =map_sha1_file_1 (the_repository, path,NULL , &mapsize);if (!map) {error_errno ("unable to mmap %s" , path); goto out;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 19/27] sha1_file: add repository argument to map_sha1_file_1
Quoted 10 lines · On Tue, 20 Feb 2018 17:54:22 -0800 Stefan Beller <[email protected]> wrote:
Add a repository argument to allow the
map_sha1_file_1caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other thanthe_repositoryyet.As with the previous commits, use a macro to catch callers passing a repository other than
the_repositoryat compile time.Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>
PATCH v3 20/27 sha1_file: add repository argument to map_sha1_file
In reply to Stefan BellerAdd a repository argument to allow map_sha1_file callers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
While at it, move the declaration to object-store.h, where it should be easier to find.
--- cache.h | 1 - object-store.h | 3 +++ sha1_file.c | 4 ++-- streaming.c | 5 ++++- 4 files changed, 9 insertions(+), 4 deletions(-)
cache.h +0 -1
@@ -1242,7 +1242,6 @@ extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned cextern int force_object_loose (const unsigned char *sha1,time_t mtime);extern int git_open_cloexec (const char *name,int flags);#define git_open (name) git_open_cloexec(name, O_RDONLY) -extern void *map_sha1_file (const unsigned char *sha1,unsigned long *size);extern int unpack_sha1_header (git_zstream *stream,unsigned char *map,unsigned long mapsize,void *buffer,unsigned long bufsiz);extern int parse_sha1_header (const char *hdr,unsigned long *sizep);
object-store.h +3 -0
@@ -130,4 +130,7 @@ void raw_object_store_clear(struct raw_object_store *o);#define sha1_file_name (r, b, s) sha1_file_name_##r(b, s)void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1); +#define map_sha1_file (r, s, sz) map_sha1_file_##r(s, sz) +void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size); +#endif /* OBJECT_STORE_H */
sha1_file.c +2 -2
@@ -968,7 +968,7 @@ static void *map_sha1_file_1_the_repository(const char *path,return map; } -void *map_sha1_file (const unsigned char *sha1,unsigned long *size) +void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size) {return map_sha1_file_1 (the_repository,NULL , sha1, size); }
@@ -1192,7 +1192,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,return 0 ; } - map =map_sha1_file (sha1, &mapsize); + map =map_sha1_file (the_repository, sha1, &mapsize);if (!map)return -1 ;
streaming.c +4 -1
@@ -3,6 +3,8 @@*/ #include "cache.h" #include "streaming.h" +#include "repository.h" +#include "object-store.h" #include "packfile.h" enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {static open_method_decl (loose) { - st->u.loose.mapped =map_sha1_file (sha1, &st->u.loose.mapsize); + st->u.loose.mapped =map_sha1_file (the_repository, + sha1, &st->u.loose.mapsize);if (!st->u.loose.mapped)return -1 ; if ((unpack_sha1_header(&st->z,
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v3 21/27 sha1_file: add repository argument to sha1_loose_object_info
In reply to Stefan BellerAdd a repository argument to allow the sha1_loose_object_info caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
sha1_file.c +5 -4
@@ -1159,9 +1159,10 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -static int sha1_loose_object_info (const unsigned char *sha1, -struct object_info *oi, -int flags) +#define sha1_loose_object_info (r, s, o, f) sha1_loose_object_info_##r(s, o, f) +static int sha1_loose_object_info_the_repository (const unsigned char *sha1, +struct object_info *oi, +int flags) {int status =0 ;unsigned long mapsize;
@@ -1280,7 +1281,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,break ;/* Most likely it's a loose object. */ -if (!sha1_loose_object_info (real, oi, flags)) +if (!sha1_loose_object_info (the_repository, real, oi, flags))return 0 ;/* Not a loose object; someone else may have just packed it. */
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 21/27] sha1_file: add repository argument to sha1_loose_object_info
Hi,
Quoted 5 lines · Stefan Beller
Add a repository argument to allow the
sha1_loose_object_infocaller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other thanthe_repositoryyet.
Quoted 8 lines
As with the previous commits, use a macro to catch callers passing a repository other than
the_repositoryat compile time.Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>--- sha1_file.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
Simple and obviously correct.
PATCH v3 22/27 sha1_file: allow sha1_file_name to handle arbitrary repositories
In reply to Stefan Beller--- object-store.h | 3 +-- sha1_file.c | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-)
object-store.h +1 -2
@@ -127,8 +127,7 @@ void raw_object_store_clear(struct raw_object_store *o);* Put in `buf` the name of the file in the local object database that * would be used to store a loose object with the specified sha1. */ -#define sha1_file_name (r, b, s) sha1_file_name_##r(b, s) -void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1); +void sha1_file_name (struct repository *r,struct strbuf *buf,const unsigned char *sha1);#define map_sha1_file (r, s, sz) map_sha1_file_##r(s, sz)void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size);
sha1_file.c +2 -2
@@ -323,9 +323,9 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1) } } -void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1) +void sha1_file_name (struct repository *r,struct strbuf *buf,const unsigned char *sha1) { -strbuf_addstr (buf,get_object_directory ()); +strbuf_addstr (buf, r->objects.objectdir);strbuf_addch (buf,'/' );fill_sha1_path (buf, sha1); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories
Quoted 2 lines · On Tue, 20 Feb 2018 17:54:25 -0800 Stefan Beller <[email protected]> wrote:
Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>
sha1_file.c static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1) · line 331
Quoted 8 lines
sha1_file.c +2
-2-void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1) +void sha1_file_name (struct repository *r,struct strbuf *buf,const unsigned char *sha1) { -strbuf_addstr (buf,get_object_directory ()); +strbuf_addstr (buf, r->objects.objectdir);strbuf_addch (buf,'/' );fill_sha1_path (buf, sha1); }
In the future, we should probably have:
- a function to get the object store out of a repo (so that it can lazily initialize the object store struct if necessary)
- when the object store is obtained, its objectdir field is guaranteed to be populated
- sha1_file_name should take the object store struct, not the repo struct
but this is outside the scope of this patch.
Re: [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories
Quoted 8 lines · Jonathan Tan
Quoted 2 lines · On Tue, 20 Feb 2018 17:54:25 -0800 Stefan Beller <[email protected]> wrote:
Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>Reviewed-by Jonathan Tan <[email protected]>
Quoted 16 lines
sha1_file.c static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1) · line 331
Quoted 8 lines
sha1_file.c +2
-2-void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1) +void sha1_file_name (struct repository *r,struct strbuf *buf,const unsigned char *sha1) { -strbuf_addstr (buf,get_object_directory ()); +strbuf_addstr (buf, r->objects.objectdir);strbuf_addch (buf,'/' );fill_sha1_path (buf, sha1); }In the future, we should probably have:
- a function to get the object store out of a repo (so that it can lazily initialize the object store struct if necessary)
- when the object store is obtained, its objectdir field is guaranteed to be populated
-sha1_file_nameshould take the object store struct, not the repo struct
We had that in v2, but I reverted it as the consensus seemed that we prefer the_repository to be passed around.
Quoted 2 lines
but this is outside the scope of this patch.
PATCH v3 23/27 sha1_file: allow stat_sha1_file to handle arbitrary repositories
In reply to Stefan Beller--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -874,23 +874,22 @@ int git_open_cloexec(const char *name, int flags)* Note that it may point to static storage and is only valid until another * call to sha1_file_name(), etc. */ -#define stat_sha1_file (r, s, st, p) stat_sha1_file_##r(s, st, p) -static int stat_sha1_file_the_repository (const unsigned char *sha1, -struct stat *st,const char **path) +static int stat_sha1_file (struct repository *r,const unsigned char *sha1, +struct stat *st,const char **path) {struct alternate_object_database *alt;static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (the_repository, &buf, sha1); +sha1_file_name (r, &buf, sha1); *path = buf.buf;if (!lstat (*path, st))return 0 ; -prepare_alt_odb (the_repository); +prepare_alt_odb (r); errno =ENOENT ; -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +for (alt = r->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);if (!lstat (*path, st))return 0 ;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v3 24/27 sha1_file: allow open_sha1_file to handle arbitrary repositories
In reply to Stefan Beller--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -902,9 +902,8 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,* Like stat_sha1_file(), but actually open the object and return the * descriptor. See the caveats on the "path" parameter above. */ -#define open_sha1_file (r, s, p) open_sha1_file_##r(s, p) -static int open_sha1_file_the_repository (const unsigned char *sha1, -const char **path) +static int open_sha1_file (struct repository *r, +const unsigned char *sha1,const char **path) {int fd;struct alternate_object_database *alt;
@@ -912,7 +911,7 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (the_repository, &buf, sha1); +sha1_file_name (r, &buf, sha1); *path = buf.buf; fd =git_open (*path);
@@ -920,8 +919,8 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,return fd; most_interesting_errno = errno; -prepare_alt_odb (the_repository); -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +prepare_alt_odb (r); +for (alt = r->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);if (fd >=0 )
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 24/27] sha1_file: allow open_sha1_file to handle arbitrary repositories
Quoted 2 lines · On Tue, 20 Feb 2018 17:54:27 -0800 Stefan Beller <[email protected]> wrote:
Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>
PATCH v3 25/27 sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
--- sha1_file.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
sha1_file.c +3 -5
@@ -936,10 +936,8 @@ static int open_sha1_file(struct repository *r,* Map the loose object at "path" if it is not NULL, or the path found by * searching for a loose object named "sha1". */ -#define map_sha1_file_1 (r, p, s, si) map_sha1_file_1_##r(p, s, si) -static void *map_sha1_file_1_the_repository (const char *path, -const unsigned char *sha1, -unsigned long *size) +static void *map_sha1_file_1 (struct repository *r,const char *path, +const unsigned char *sha1,unsigned long *size) {void *map;int fd;
@@ -947,7 +945,7 @@ static void *map_sha1_file_1_the_repository(const char *path,if (path) fd =git_open (path);else - fd =open_sha1_file (the_repository, sha1, &path); + fd =open_sha1_file (r, sha1, &path); map =NULL ;if (fd >=0 ) {struct stat st;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v3 26/27 sha1_file: allow map_sha1_file to handle arbitrary repositories
In reply to Stefan Beller--- object-store.h | 3 +-- sha1_file.c | 5 +++-- 2 files changed, 4 insertions(+), 4 deletions(-)
object-store.h +1 -2
@@ -129,7 +129,6 @@ void raw_object_store_clear(struct raw_object_store *o);*/ void sha1_file_name (struct repository *r,struct strbuf *buf,const unsigned char *sha1); -#define map_sha1_file (r, s, sz) map_sha1_file_##r(s, sz) -void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size); +void *map_sha1_file (struct repository *r,const unsigned char *sha1,unsigned long *size);#endif /* OBJECT_STORE_H */
sha1_file.c +3 -2
@@ -964,9 +964,10 @@ static void *map_sha1_file_1(struct repository *r, const char *path,return map; } -void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size) +void *map_sha1_file (struct repository *r, +const unsigned char *sha1,unsigned long *size) { -return map_sha1_file_1 (the_repository,NULL , sha1, size); +return map_sha1_file_1 (r,NULL , sha1, size); } static int unpack_sha1_short_header(git_zstream *stream,
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCH 26/27] sha1_file: allow map_sha1_file to handle arbitrary repositories
Quoted 6 lines · Stefan Beller
Signed-off-by Stefan Beller <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>--- object-store.h | 3 +-- sha1_file.c | 5 +++-- 2 files changed, 4 insertions(+), 4 deletions(-)
Thanks: this is short and simple, making my life as a reviewer very easy.
PATCH v3 27/27 sha1_file: allow sha1_loose_object_info to handle arbitrary repositories
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -1156,10 +1156,9 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -#define sha1_loose_object_info (r, s, o, f) sha1_loose_object_info_##r(s, o, f) -static int sha1_loose_object_info_the_repository (const unsigned char *sha1, -struct object_info *oi, -int flags) +static int sha1_loose_object_info (struct repository *r, +const unsigned char *sha1, +struct object_info *oi,int flags) {int status =0 ;unsigned long mapsize;
@@ -1183,14 +1182,14 @@ static int sha1_loose_object_info_the_repository(const unsigned char *sha1,if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {const char *path;struct stat st; -if (stat_sha1_file (the_repository, sha1, &st, &path) <0 ) +if (stat_sha1_file (r, sha1, &st, &path) <0 )return -1 ;if (oi->disk_sizep) *oi->disk_sizep = st.st_size;return 0 ; } - map =map_sha1_file (the_repository, sha1, &mapsize); + map =map_sha1_file (r, sha1, &mapsize);if (!map)return -1 ;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Quoted 8 lines · Stefan Beller
v3:
* reverted back to use the repository for most of the functions (including unduplicating 'ignore_env')
* rebased on master again (I lost that state when doing v2, as I did both rebase as well as conversion to object store in one go)
* one additional patch, that moves the alternates related things toobject-store.h, such that inclusion ofcache.h(inobject-store.h) is not required any more.
This is also available at
https://github.com/stefanbeller/git/tree/object-store-part1-v3
v4 cover Moving global state into the repository object (part 1)
In reply to Stefan Bellerv4:
* addressed feedback from the single patches (mostly nits)
* rebased on latest master
v3:
* reverted back to use the repository for most of the functions (including unduplicating 'ignore_env')
* rebased on master again (I lost that state when doing v2, as I did both rebase as well as conversion to object store in one go)
* one additional patch, that moves the alternates related things to object-store.h, such that inclusion of cache.h (in object-store.h) is not required any more.
v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global, which means there is just one patch per function that is converted. As this follows the same structure of the previous series, I am still confident there is no hidden dependencies to globals outside the object store in these converted functions.
* rebased on top of current master, resolving the merge conflicts.
I think I used the list.h APIs right, but please double check.
Thanks,
Stefan
v1:
This is a real take on the first part of the recent RFC[1].
Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories" might be a good breaking point for a first part at that RFC at patch 38. This series is smaller and contains only 26 patches as the patches in the big RFC were slightly out of order.
I developed this series partly by writing patches, but mostly by cherrypicking from that RFC on top of current master. I noticed no external conflicts apart from one addition to the repositories _INIT macro, which was easy to resolve.
Comments in the early range of that RFC were on 003 where Junio pointed out that the coccinelle patch ought to be not in contrib/coccinelle, so I put it in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.
brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself without any suggestion to include into this series[3].
Duy suggested that we shall not use the repository blindly, but should carefully examine whether to pass on an object store or the refstore or such[4], which I agree with if it makes sense. This series unfortunately has an issue with that as I would not want to pass down the ignore_env flag separately from the object store, so I made all functions that only take the object store to have the raw object store as the first parameter, and others using the full repository.
Eric Sunshine brought up memory leaks with the RFC, and I would think to have plugged all holes.
[1] https://public-inbox.org/git/[email protected]/
[2] https://public-inbox.org/git/[email protected]/
[3] https://public-inbox.org/git/[email protected]/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/
Thanks,
Stefan
Jonathan Nieder (2):
sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
sha1_file: allow sha1_loose_object_info to handle arbitrary
repositories
Stefan Beller (25):
repository: introduce raw object store field
object-store: migrate alternates struct and functions from cache.h
object-store: move alt_odb_list and alt_odb_tail to object store
object-store: free alt_odb_list
object-store: move packed_git and packed_git_mru to object store
object-store: close all packs upon clearing the object store
pack: move prepare_packed_git_run_once to object store
pack: move approximate object count to object store
sha1_file: add raw_object_store argument to alt_odb_usable
sha1_file: add repository argument to link_alt_odb_entry
sha1_file: add repository argument to read_info_alternates
sha1_file: add repository argument to link_alt_odb_entries
sha1_file: add repository argument to prepare_alt_odb
sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
sha1_file: allow prepare_alt_odb to handle arbitrary repositories
sha1_file: add repository argument to sha1_file_name
sha1_file: add repository argument to stat_sha1_file
sha1_file: add repository argument to open_sha1_file
sha1_file: add repository argument to map_sha1_file_1
sha1_file: add repository argument to map_sha1_file
sha1_file: add repository argument to sha1_loose_object_info
sha1_file: allow sha1_file_name to handle arbitrary repositories
sha1_file: allow stat_sha1_file to handle arbitrary repositories
sha1_file: allow open_sha1_file to handle arbitrary repositories
sha1_file: allow map_sha1_file to handle arbitrary repositories
builtin/am.c | 2 +- builtin/clone.c | 2 +- builtin/count-objects.c | 5 +- builtin/fetch.c | 2 +- builtin/fsck.c | 12 ++-- builtin/gc.c | 3 +- builtin/grep.c | 2 +- builtin/merge.c | 2 +- builtin/pack-objects.c | 20 ++++--- builtin/pack-redundant.c | 5 +- builtin/receive-pack.c | 2 +- cache.h | 87 --------------------------- environment.c | 5 +- fast-import.c | 7 ++- http-backend.c | 5 +- http-walker.c | 3 +- http.c | 5 +- object-store.h | 124 ++++++++++++++++++++++++++++++++++++++ object.c | 27 +++++++++ pack-bitmap.c | 3 +- packfile.c | 64 ++++++++++---------- packfile.h | 2 +- path.c | 2 +- repository.c | 21 +++++-- repository.h | 7 ++- server-info.c | 5 +- sha1_file.c | 125 +++++++++++++++++++++------------------ sha1_name.c | 10 ++-- streaming.c | 5 +- 29 files changed, 335 insertions(+), 229 deletions(-) create mode 100644 object-store.h
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 1/27 repository: introduce raw object store field
The raw object store field will contain any objects needed for access to objects in a given repository.
This patch introduces the raw object store and populates it with the objectdir, which used to be part of the repository struct.
As the struct gains members, we'll also populate the function to clear the memory for these members.
In a later we'll introduce a struct object_parser, that will complement the object parsing in a repository struct: The raw object parser is the layer that will provide access to raw object content, while the higher level object parser code will parse raw objects and keeps track of parenthood and other object relationships using 'struct object'. For now only add the lower level to the repository struct.
--- builtin/grep.c | 2 +- environment.c | 5 +++-- object-store.h | 15 +++++++++++++++ object.c | 5 +++++ path.c | 2 +- repository.c | 19 ++++++++++++++----- repository.h | 7 ++++--- 7 files changed, 43 insertions(+), 12 deletions(-) create mode 100644 object-store.h
builtin/grep.c +1 -1
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,* object. */ grep_read_lock (); -add_to_alternates_memory (submodule.objectdir); +add_to_alternates_memory (submodule.objects.objectdir);grep_read_unlock ();if (oid) {
environment.c +3 -2
@@ -13,6 +13,7 @@#include "refs.h" #include "fmt-merge-msg.h" #include "commit.h" +#include "object-store.h" int trust_executable_bit =1 ;int trust_ctime =1 ;
@@ -244,9 +245,9 @@ const char *get_git_work_tree(void)char *get_object_directory (void ) { -if (!the_repository->objectdir) +if (!the_repository->objects.objectdir)BUG ("git environment hasn't been setup" ); -return the_repository->objectdir; +return the_repository->objects.objectdir; }int odb_mkstemp (struct strbuf *template,const char *pattern)
object-store.h new file +15 -0
@@ -0,0 +1,15 @@ +#ifndef OBJECT_STORE_H +#define OBJECT_STORE_H + +struct raw_object_store { +/* +* Path to the repository's object store. +* Cannot be NULL after initialization. +*/ +char *objectdir; +}; +#define RAW_OBJECT_STORE_INIT { NULL } + +void raw_object_store_clear (struct raw_object_store *o); + +#endif /* OBJECT_STORE_H */
object.c +5 -0
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags) obj->flags &= ~flags; } } + +void raw_object_store_clear (struct raw_object_store *o) +{ +FREE_AND_NULL (o->objectdir); +}
path.c +1 -1
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,strbuf_splice (buf,0 , buf->len, repo->index_file,strlen (repo->index_file)); else if (dir_prefix(base, "objects")) - replace_dir(buf, git_dir_len + 7, repo->objectdir); + replace_dir(buf, git_dir_len + 7, repo->objects.objectdir); else if (git_hooks_path && dir_prefix(base, "hooks")) replace_dir(buf, git_dir_len + 5, git_hooks_path); else if (repo->different_commondir)
repository.c +14 -5
@@ -1,11 +1,18 @@#include "cache.h" #include "repository.h" +#include "object-store.h" #include "config.h" #include "submodule-config.h" /* The main repository */ static struct repository the_repo = { -NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index, &hash_algos[GIT_HASH_SHA1 ],0 ,0 +NULL ,NULL , +RAW_OBJECT_STORE_INIT , +NULL ,NULL ,NULL , +NULL ,NULL ,NULL , + &the_index, + &hash_algos[GIT_HASH_SHA1 ], +0 ,0 };struct repository *the_repository = &the_repo;
@@ -42,9 +49,10 @@ static void repo_setup_env(struct repository *repo) !repo->ignore_env);free (repo->commondir); repo->commondir =strbuf_detach (&sb,NULL ); -free (repo->objectdir); - repo->objectdir =git_path_from_env (DB_ENVIRONMENT , repo->commondir, -"objects" , !repo->ignore_env); +raw_object_store_clear (&repo->objects); + repo->objects.objectdir = +git_path_from_env (DB_ENVIRONMENT , repo->commondir, +"objects" , !repo->ignore_env);free (repo->graft_file); repo->graft_file =git_path_from_env (GRAFT_ENVIRONMENT , repo->commondir,"info/grafts" , !repo->ignore_env);
@@ -209,12 +217,13 @@ void repo_clear(struct repository *repo) {FREE_AND_NULL (repo->gitdir);FREE_AND_NULL (repo->commondir); -FREE_AND_NULL (repo->objectdir);FREE_AND_NULL (repo->graft_file);FREE_AND_NULL (repo->index_file);FREE_AND_NULL (repo->worktree);FREE_AND_NULL (repo->submodule_prefix); +raw_object_store_clear (&repo->objects); +if (repo->config) {git_configset_clear (repo->config);FREE_AND_NULL (repo->config);
repository.h +4 -3
@@ -1,6 +1,8 @@#ifndef REPOSITORY_H #define REPOSITORY_H +#include "object-store.h" +struct config_set ;struct index_state ;struct submodule_cache ;
@@ -21,10 +23,9 @@ struct repository {char *commondir;/* -* Path to the repository's object store. -* Cannot be NULL after initialization. +* Holds any information related to accessing the raw object content. */ -char *objectdir; +struct raw_object_store objects;/* * Path to the repository's graft file.
Signature · 1 line
2.16.1.291.g4437f3f132-goog
Re: [PATCHv4 01/27] repository: introduce raw object store field
Review
Quoted 12 lines · Stefan Beller
/* The main repository */
static struct repositorythe_repo= {(quoted diff) +7
-1- NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0 + NULL, NULL, + RAW_OBJECT_STORE_INIT, + NULL, NULL, NULL, + NULL, NULL, NULL, + &the_index, + &hash_algos[GIT_HASH_SHA1], + 0, 0 }; struct repository *the_repository = &the_repo;
I wonder if we should do something like this. It makes the_repo initialization easier to read and it helps unify the init code with submodule.
No don't reroll. If you think it's a good idea, you can do something like this in the next series instead.
-- 8< --
check-racy.c +2 -0
@@ -1,10 +1,12 @@#include "cache.h" +#include "repository.h" int main (int ac,char **av) {int i;int dirty, clean, racy; +init_the_repository (); dirty = clean = racy =0 ;read_cache ();for (i =0 ; i < active_nr; i++) {
common-main.c +3 -0
@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the
@@ -32,6 +33,8 @@ int main(int argc, const char **argv)*/ sanitize_stdfds (); +init_the_repository (); +git_setup_gettext ();attr_start ();
object-store.h +1 -1
@@ -8,8 +8,8 @@ struct raw_object_store {*/ char *objectdir; }; -#define RAW_OBJECT_STORE_INIT { NULL } +void raw_object_store_init (struct raw_object_store *o);void raw_object_store_clear (struct raw_object_store *o);#endif /* OBJECT_STORE_H */
object.c +5 -0
@@ -446,6 +446,11 @@ void clear_commit_marks_all(unsigned int flags) } } +void raw_object_store_init (struct raw_object_store *o) +{ +memset (o,0 ,sizeof (*o)); +} +void raw_object_store_clear (struct raw_object_store *o) {free (o->objectdir);
repository.c +18 -11
@@ -5,16 +5,22 @@#include "submodule-config.h" /* The main repository */ -static struct repository the_repo = { -NULL ,NULL , -RAW_OBJECT_STORE_INIT , -NULL ,NULL ,NULL , -NULL ,NULL ,NULL , - &the_index, - &hash_algos[GIT_HASH_SHA1 ], -0 ,0 -}; -struct repository *the_repository = &the_repo; +static struct repository the_repo; +struct repository *the_repository; + +static void repo_pre_init (struct repository *repo) +{ +memset (repo,0 ,sizeof (*repo)); +raw_object_store_init (&repo->objects); +} + +void init_the_repository (void ) +{ + the_repository = &the_repo; +repo_pre_init (the_repository); + the_repository->index = &the_index; +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +}static char *git_path_from_env (const char *envvar,const char *git_dir,const char *path,int fromenv)
@@ -138,7 +144,8 @@ static int read_and_verify_repository_format(struct repository_format *format,int repo_init (struct repository *repo,const char *gitdir,const char *worktree) {struct repository_format format; -memset (repo,0 ,sizeof (*repo)); + +repo_pre_init (repo); repo->ignore_env =1 ;
repository.h +1 -0
@@ -93,6 +93,7 @@ extern void repo_set_gitdir(struct repository *repo, const char *path);extern void repo_set_worktree (struct repository *repo,const char *path);extern void repo_set_hash_algo (struct repository *repo,int algo);extern int repo_init (struct repository *repo,const char *gitdir,const char *worktree); +extern void init_the_repository (void );extern int repo_submodule_init (struct repository *submodule,struct repository *superproject,const char *path);
-- 8< --
Re: [PATCHv4 01/27] repository: introduce raw object store field
Quoted 17 lines · Duy Nguyen
common-main.c +2
-0@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the @@ -32,6 +33,8 @@ int main(int argc, const char **argv)*/ sanitize_stdfds (); +init_the_repository ();
common-main.c int main(int argc, const char **argv)
Quoted 10 lines
common-main.c +8
-0+git_setup_gettext (); +void init_the_repository (void ) +{ + the_repository = &the_repo; +repo_pre_init (the_repository); + the_repository->index = &the_index; +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +}
I see what you did here, and I like it.
Re: [PATCHv4 01/27] repository: introduce raw object store field
Quoted 18 lines · Junio C Hamano
Quoted 16 lines · Duy Nguyen
common-main.c +1
-0@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the @@ -32,6 +33,8 @@ int main(int argc, const char **argv)*/ sanitize_stdfds ();
Quoted 13 lines
common-main.c int main(int argc, const char **argv)
Quoted 11 lines
common-main.c +9
-0+init_the_repository (); +git_setup_gettext (); +void init_the_repository (void ) +{ + the_repository = &the_repo; +repo_pre_init (the_repository); + the_repository->index = &the_index; +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +}I see what you did here, and I like it.
I thought this would be a good idea to do eventually but back when I first introduced struct repository there wasn't enough to justify it till now. This definitely makes it much easier to read the initialization and I prefer this over the initializer. Thanks for working on this :)
Signature · 1 line
Brandon Williams
Re: [PATCHv4 01/27] repository: introduce raw object store field
In reply to Duy Nguyencheck-racy.c
Quoted 26 lines · Duy Nguyen
Review
Quoted 12 lines · Stefan Beller
/* The main repository */
static struct repositorythe_repo= {(quoted diff) +7
-1- NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0 + NULL, NULL, + RAW_OBJECT_STORE_INIT, + NULL, NULL, NULL, + NULL, NULL, NULL, + &the_index, + &hash_algos[GIT_HASH_SHA1], + 0, 0 }; struct repository *the_repository = &the_repo;I wonder if we should do something like this. It makes
the_repoinitialization easier to read and it helps unify the init code with submodule.No don't reroll. If you think it's a good idea, you can do something like this in the next series instead.
-- 8< --
check-racy.c +0
-0
totally offtopic: Do we want to move this file into t/helper?
Quoted 11 lines
common-main.c +1
-0@@ -1,6 +1,7 @@#include "cache.h" #include "exec_cmd.h" #include "attr.h" +#include "repository.h" /* * Many parts of Git have subprograms communicate via pipe, expect the
common-main.c int main(int argc, const char **argv) · line 37
Quoted 5 lines
common-main.c +2
-0@@ -32,6 +33,8 @@ int main(int argc, const char **argv)*/ sanitize_stdfds (); +init_the_repository (); +
So this is the real deal.
Quoted 11 lines
common-main.c +1
-1-#define RAW_OBJECT_STORE_INIT { NULL } +void raw_object_store_init (struct raw_object_store *o);void raw_object_store_clear (struct raw_object_store *o);#endif /*
OBJECT_STORE_H*/object.c +0
-0
object.c void clear_commit_marks_all(unsigned int flags) · line 452
Quoted 7 lines
object.c +4
-0@@ -446,6 +446,11 @@ void clear_commit_marks_all(unsigned int flags) } } +void raw_object_store_init (struct raw_object_store *o) +{ +memset (o,0 ,sizeof (*o)); +}
We'll have to adapt this for the list that we use for packed_git_mru, but that should be no problem.
Quoted 18 lines
object.c +14
-0+ +static void repo_pre_init (struct repository *repo) +{ +memset (repo,0 ,sizeof (*repo)); +raw_object_store_init (&repo->objects); +} + +void init_the_repository (void ) +{ + the_repository = &the_repo; +repo_pre_init (the_repository); + the_repository->index = &the_index; +repo_set_hash_algo (the_repository,GIT_HASH_SHA1 ); +}static char *git_path_from_env (const char *envvar,const char *git_dir,const char *path,int fromenv)
object.c static int read_and_verify_repository_format(struct repository_format *format, · line 151
Quoted 9 lines
object.c +2
-1@@ -138,7 +144,8 @@ static int read_and_verify_repository_format(struct repository_format *format,int repo_init (struct repository *repo,const char *gitdir,const char *worktree) {struct repository_format format; -memset (repo,0 ,sizeof (*repo)); + +repo_pre_init (repo); repo->ignore_env =1 ;
I like the approach. Though as you say, let's have it in the next series.
Thanks for the thoughts and guidance,
Stefan
Re: [PATCHv4 01/27] repository: introduce raw object store field
Quoted 22 lines · Stefan Beller
Quoted 21 lines · Duy Nguyen
Review
Quoted 12 lines · Stefan Beller
/* The main repository */
static struct repositorythe_repo= {(quoted diff) +7
-1- NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0 + NULL, NULL, + RAW_OBJECT_STORE_INIT, + NULL, NULL, NULL, + NULL, NULL, NULL, + &the_index, + &hash_algos[GIT_HASH_SHA1], + 0, 0 }; struct repository *the_repository = &the_repo;I wonder if we should do something like this. It makes
the_repoinitialization easier to read and it helps unify the init code with submodule.No don't reroll. If you think it's a good idea, you can do something like this in the next series instead.
Quoted 7 lines
check-racy.c
Quoted 5 lines
-- 8< --
check-racy.c +0
-0totally offtopic: Do we want to move this file into t/helper?
No wonder both Jeff and I missed this program (he didn't convert it to use cmd_main, and I didn't move it to t/helper). This git-check-racy is added in 42f774063d (Add check program "git-check-racy" - 2006-08-15) and is not part of the default build. You need to manually update Makefile first to build it.
Right now it's broken (multiple definition of 'main'). If you add init_the_repository() or something similar, just leave this file untouched. Maybe I'll fix it later, separately. Or perhaps I'll move this functionality to git-update-index if this is still worth keeping.
Signature · 1 line
Duy
PATCH v4 2/27 object-store: migrate alternates struct and functions from cache.h
In reply to Stefan BellerMigrate the struct alternate_object_database and all its related functions to the object store as these functions are easier found in that header. The migration is just a verbatim copy, no need to include the object store header at any C file, because cache.h includes repository.h which in turn includes the object-store.h
--- cache.h | 51 -------------------------------------------------- object-store.h | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 51 deletions(-)
cache.h +0 -51
@@ -1560,57 +1560,6 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);extern void schedule_dir_for_removal (const char *name,int len);extern void remove_scheduled_dirs (void ); -extern struct alternate_object_database { -struct alternate_object_database *next; - -/* see alt_scratch_buf() */ -struct strbuf scratch; -size_t base_len; - -/* -* Used to store the results of readdir(3) calls when searching -* for unique abbreviated hashes. This cache is never -* invalidated, thus it's racy and not necessarily accurate. -* That's fine for its purpose; don't use it for tasks requiring -* greater accuracy! -*/ -char loose_objects_subdir_seen[256 ]; -struct oid_array loose_objects_cache; - -char path[FLEX_ARRAY ]; -} *alt_odb_list; -extern void prepare_alt_odb (void ); -extern char *compute_alternate_path (const char *path,struct strbuf *err); -typedef int alt_odb_fn (struct alternate_object_database *,void *); -extern int foreach_alt_odb (alt_odb_fn ,void *); - -/* -* Allocate a "struct alternate_object_database" but do _not_ actually -* add it to the list of alternates. -*/ -struct alternate_object_database *alloc_alt_odb (const char *dir); - -/* -* Add the directory to the on-disk alternates file; the new entry will also -* take effect in the current process. -*/ -extern void add_to_alternates_file (const char *dir); - -/* -* Add the directory to the in-memory list of alternates (along with any -* recursive alternates it points to), but do not modify the on-disk alternates -* file. -*/ -extern void add_to_alternates_memory (const char *dir); - -/* -* Returns a scratch strbuf pre-filled with the alternate object directory, -* including a trailing slash, which can be used to access paths in the -* alternate. Always use this over direct access to alt->scratch, as it -* cleans up any previous use of the scratch buffer. -*/ -extern struct strbuf *alt_scratch_buf (struct alternate_object_database *alt); -struct pack_window {struct pack_window *next;unsigned char *base;
object-store.h +51 -0
57 lines not on this page · raw message
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 3/27 object-store: move alt_odb_list and alt_odb_tail to object store
In reply to Stefan BellerIn a process with multiple repositories open, alternates should be associated to a single repository and not shared globally. Move alt_odb_list and alt_odb_tail into the_repository and adjust callers to reflect this.
Now that the alternative object data base is per repository, we're leaking its memory upon freeing a repository. The next patch plugs this hole.
No functional change intended.
--- builtin/fsck.c | 4 +++- object-store.h | 9 ++++++--- packfile.c | 3 ++- sha1_file.c | 25 ++++++++++++------------- sha1_name.c | 3 ++- 5 files changed, 25 insertions(+), 19 deletions(-)
builtin/fsck.c +3 -1
@@ -1,5 +1,6 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "commit.h" #include "tree.h"
@@ -722,7 +723,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)fsck_object_dir (get_object_directory ());prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; + alt; alt = alt->next)fsck_object_dir (alt->path);if (check_full) {
object-store.h +6 -3
@@ -1,7 +1,7 @@ #ifndef OBJECT_STORE_H #define OBJECT_STORE_H -extern struct alternate_object_database { +struct alternate_object_database { struct alternate_object_database *next; /* see alt_scratch_buf() */
@@ -19,7 +19,7 @@ extern struct alternate_object_database {struct oid_array loose_objects_cache;char path[FLEX_ARRAY ]; -} *alt_odb_list; +};void prepare_alt_odb (void );char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);
@@ -58,8 +58,11 @@ struct raw_object_store {* Cannot be NULL after initialization. */ char *objectdir; + +struct alternate_object_database *alt_odb_list; +struct alternate_object_database **alt_odb_tail; }; -#define RAW_OBJECT_STORE_INIT { NULL } +#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +2 -1
@@ -1,6 +1,7 @@#include "cache.h" #include "list.h" #include "pack.h" +#include "repository.h" #include "dir.h" #include "mergesort.h" #include "packfile.h"
@@ -891,7 +892,7 @@ void prepare_packed_git(void)return ;prepare_packed_git_one (get_object_directory (),1 );prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();prepare_packed_git_mru ();
sha1_file.c +12 -13
@@ -22,6 +22,7 @@#include "pack-revindex.h" #include "sha1-lookup.h" #include "bulk-checkin.h" +#include "repository.h" #include "streaming.h" #include "dir.h" #include "list.h"
@@ -340,9 +341,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,return buf->buf; } -struct alternate_object_database *alt_odb_list; -static struct alternate_object_database **alt_odb_tail; -/* * Return non-zero iff the path is usable as an alternate object database. */
@@ -362,7 +360,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)* Prevent the common mistake of listing the same * thing twice, or object directory itself. */ -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {if (!fspathcmp (path->buf, alt->path))return 0 ; }
@@ -422,8 +420,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base, ent =alloc_alt_odb (pathbuf.buf);/* add the alternate entry */ - *alt_odb_tail = ent; - alt_odb_tail = &(ent->next); + *the_repository->objects.alt_odb_tail = ent; + the_repository->objects.alt_odb_tail = &(ent->next); ent->next =NULL ;/* recursively add alternates */
@@ -557,7 +555,7 @@ void add_to_alternates_file(const char *reference)fprintf_or_die (out,"%s\n" , reference);if (commit_lock_file (&lock))die_errno ("unable to move new alternates file into place" ); -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)link_alt_odb_entries (reference,'\n' ,NULL ,0 ); }free (alts);
@@ -655,7 +653,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)int r =0 ;prepare_alt_odb (); -for (ent = alt_odb_list; ent; ent = ent->next) { +for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) { r =fn (ent, cb);if (r)break ;
@@ -667,12 +665,13 @@ void prepare_alt_odb(void) {const char *alt; -if (alt_odb_tail) +if (the_repository->objects.alt_odb_tail)return ; alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - alt_odb_tail = &alt_odb_list; + the_repository->objects.alt_odb_tail = + &the_repository->objects.alt_odb_list;link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 );read_info_alternates (get_object_directory (),0 );
@@ -716,7 +715,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen) {struct alternate_object_database *alt;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {const char *path =alt_sha1_path (alt, sha1);if (check_and_freshen_file (path, freshen))return 1 ;
@@ -876,7 +875,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,prepare_alt_odb (); errno =ENOENT ; -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);if (!lstat (*path, st))return 0 ;
@@ -906,7 +905,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path) most_interesting_errno = errno;prepare_alt_odb (); -for (alt = alt_odb_list; alt; alt = alt->next) { +for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);if (fd >=0 )
sha1_name.c +2 -1
@@ -10,6 +10,7 @@#include "dir.h" #include "sha1-array.h" #include "packfile.h" +#include "repository.h" static int get_oid_oneline (const char *,struct object_id *,struct commit_list *);
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)*/ fakeent =alloc_alt_odb (get_object_directory ()); } - fakeent->next = alt_odb_list; + fakeent->next = the_repository->objects.alt_odb_list;for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {int pos;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 4/27 object-store: free alt_odb_list
In reply to Stefan BellerFree the memory and reset alt_odb_{list, tail} to NULL.
--- object.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
object.c +17 -0
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags) } } +static void free_alt_odb (struct alternate_object_database *alt) +{ +strbuf_release (&alt->scratch); +oid_array_clear (&alt->loose_objects_cache); +} + +static void free_alt_odbs (struct raw_object_store *o) +{ +while (o->alt_odb_list) { +free_alt_odb (o->alt_odb_list); + o->alt_odb_list = o->alt_odb_list->next; + } +} +void raw_object_store_clear (struct raw_object_store *o) {FREE_AND_NULL (o->objectdir); + +free_alt_odbs (o); + o->alt_odb_tail =NULL ; }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 5/27 object-store: move packed_git and packed_git_mru to object store
In reply to Stefan BellerIn a process with multiple repositories open, packfile accessors should be associated to a single repository and not shared globally. Move packed_git and packed_git_mru into the_repository and adjust callers to reflect this.
Patch generated by
1. Moving the struct packed_git declaration to object-store.h and packed_git, packed_git_mru globals to struct object_store.
2. Apply the following semantic patch to adjust callers:
@@ @@
- packed_git
+ the_repository->objects.packed_git
@@ @@
- packed_git_mru
+ the_repository->objects.packed_git_mru
3. Applying line wrapping fixes from "make style" to break the resulting long lines.
4. Adding missing #includes of repository.h where needed.
5. As the packfiles are now owned by an objectstore, which is ephemeral unlike globals, we introduce memory leaks. So address them in raw_object_store_clear(). Defer freeing packed_git to the next patch due to the size of this patch.
--- builtin/count-objects.c | 5 +++-- builtin/fsck.c | 6 ++++-- builtin/gc.c | 3 ++- builtin/pack-objects.c | 20 +++++++++++--------- builtin/pack-redundant.c | 5 +++-- cache.h | 29 ----------------------------- fast-import.c | 7 +++++-- http-backend.c | 5 +++-- object-store.h | 31 ++++++++++++++++++++++++++++++- object.c | 6 ++++++ pack-bitmap.c | 3 ++- packfile.c | 39 ++++++++++++++++++++------------------- repository.c | 4 +++- server-info.c | 5 +++-- sha1_name.c | 5 +++-- 15 files changed, 98 insertions(+), 75 deletions(-)
builtin/count-objects.c +3 -2
@@ -7,6 +7,7 @@#include "cache.h" #include "config.h" #include "dir.h" +#include "repository.h" #include "builtin.h" #include "parse-options.h" #include "quote.h"
@@ -120,9 +121,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)struct strbuf loose_buf =STRBUF_INIT ;struct strbuf pack_buf =STRBUF_INIT ;struct strbuf garbage_buf =STRBUF_INIT ; -if (!packed_git) +if (!the_repository->objects.packed_git)prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ;if (open_pack_index (p))
builtin/fsck.c +4 -2
@@ -735,7 +735,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)prepare_packed_git ();if (show_progress) { -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {if (open_pack_index (p))continue ; total += p->num_objects;
@@ -743,7 +744,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) progress =start_progress (_ ("Checking objects" ), total); } -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {/* verify gives error messages itself */ if (verify_pack (p, fsck_obj_buffer, progress, count))
builtin/gc.c +2 -1
@@ -11,6 +11,7 @@*/ #include "builtin.h" +#include "repository.h" #include "config.h" #include "tempfile.h" #include "lockfile.h"
@@ -173,7 +174,7 @@ static int too_many_packs(void)return 0 ;prepare_packed_git (); -for (cnt =0 , p = packed_git; p; p = p->next) { +for (cnt =0 , p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ;if (p->pack_keep)
builtin/pack-objects.c +11 -9
@@ -1,5 +1,6 @@#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "attr.h" #include "object.h"
@@ -1025,8 +1026,7 @@ static int want_object_in_pack(const struct object_id *oid,if (want !=-1 )return want; } - -list_for_each (pos, &packed_git_mru) { +list_for_each (pos, &the_repository->objects.packed_git_mru) { struct packed_git *p = list_entry(pos, struct packed_git, mru);off_t offset;
@@ -1044,7 +1044,8 @@ static int want_object_in_pack(const struct object_id *oid, } want =want_found_object (exclude, p);if (!exclude && want >0 ) -list_move (&p->mru, &packed_git_mru); +list_move (&p->mru, + &the_repository->objects.packed_git_mru);if (want !=-1 )return want; }
@@ -2673,7 +2674,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)memset (&in_pack,0 ,sizeof (in_pack)); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {struct object_id oid;struct object *o;
@@ -2736,7 +2737,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)static struct packed_git *last_found = (void *)1 ;struct packed_git *p; - p = (last_found != (void *)1 ) ? last_found : packed_git; + p = (last_found != (void *)1 ) ? last_found : + the_repository->objects.packed_git;while (p) { if ((!p->pack_local || p->pack_keep) &&
@@ -2745,7 +2747,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)return 1 ; }if (p == last_found) - p = packed_git; + p = the_repository->objects.packed_git;else p = p->next;if (p == last_found)
@@ -2781,7 +2783,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)uint32_t i;struct object_id oid; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local || p->pack_keep)continue ;
@@ -3152,7 +3154,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)prepare_packed_git ();if (ignore_packed_keep) {struct packed_git *p; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)if (p->pack_local && p->pack_keep)break ;if (!p)/* no keep-able packs found */
@@ -3165,7 +3167,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)* also covers non-local objects */ struct packed_git *p; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local) { have_non_local_packs =1 ;break ;
builtin/pack-redundant.c +3 -2
@@ -7,6 +7,7 @@*/ #include "builtin.h" +#include "repository.h" #include "packfile.h" #define BLKSIZE 512
@@ -571,7 +572,7 @@ static struct pack_list * add_pack(struct packed_git *p)static struct pack_list *add_pack_file (const char *filename) { -struct packed_git *p = packed_git; +struct packed_git *p = the_repository->objects.packed_git;if (strlen (filename) <40 )die ("Bad pack filename: %s" , filename);
@@ -586,7 +587,7 @@ static struct pack_list * add_pack_file(const char *filename)static void load_all (void ) { -struct packed_git *p = packed_git; +struct packed_git *p = the_repository->objects.packed_git;while (p) {add_pack (p);
cache.h +0 -29
@@ -1569,35 +1569,6 @@ struct pack_window {unsigned int inuse_cnt; }; -extern struct packed_git { -struct packed_git *next; -struct list_head mru; -struct pack_window *windows; -off_t pack_size; -const void *index_data; -size_t index_size; -uint32_t num_objects; -uint32_t num_bad_objects; -unsigned char *bad_object_sha1; -int index_version; -time_t mtime; -int pack_fd; -unsigned pack_local:1 , - pack_keep:1 , - freshened:1 , - do_not_close:1 , - pack_promisor:1 ; -unsigned char sha1[20 ]; -struct revindex_entry *revindex; -/* something like ".git/objects/pack/xxxxx.pack" */ -char pack_name[FLEX_ARRAY ];/* more */ -} *packed_git; - -/* -* A most-recently-used ordered version of the packed_git list. -*/ -extern struct list_head packed_git_mru; -struct pack_entry {off_t offset;unsigned char sha1[20 ];
fast-import.c +5 -2
@@ -154,6 +154,7 @@ Format of STDIN stream:#include "builtin.h" #include "cache.h" +#include "repository.h" #include "config.h" #include "lockfile.h" #include "object.h"
@@ -1110,7 +1111,8 @@ static int store_object(if (e->idx.offset) { duplicate_count_by_type[type]++;return 1 ; - }else if (find_sha1_pack (oid.hash, packed_git)) { + }else if (find_sha1_pack (oid.hash, + the_repository->objects.packed_git)) { e->type = type; e->pack_id =MAX_PACK_ID ; e->idx.offset =1 ;/* just not zero! */
@@ -1307,7 +1309,8 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark) duplicate_count_by_type[OBJ_BLOB ]++;truncate_pack (&checkpoint); - }else if (find_sha1_pack (oid.hash, packed_git)) { + }else if (find_sha1_pack (oid.hash, + the_repository->objects.packed_git)) { e->type =OBJ_BLOB ; e->pack_id =MAX_PACK_ID ; e->idx.offset =1 ;/* just not zero! */
http-backend.c +3 -2
@@ -1,5 +1,6 @@#include "cache.h" #include "config.h" +#include "repository.h" #include "refs.h" #include "pkt-line.h" #include "object.h"
@@ -518,13 +519,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)select_getanyfile (hdr);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_local) cnt++; }strbuf_grow (&buf, cnt *53 +2 ); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_local)strbuf_addf (&buf,"P %s\n" , p->pack_name + objdirlen +6 ); }
object-store.h +30 -1
@@ -52,6 +52,30 @@ void add_to_alternates_memory(const char *dir);*/ struct strbuf *alt_scratch_buf (struct alternate_object_database *alt); +struct packed_git { +struct packed_git *next; +struct list_head mru; +struct pack_window *windows; +off_t pack_size; +const void *index_data; +size_t index_size; +uint32_t num_objects; +uint32_t num_bad_objects; +unsigned char *bad_object_sha1; +int index_version; +time_t mtime; +int pack_fd; +unsigned pack_local:1 , + pack_keep:1 , + freshened:1 , + do_not_close:1 , + pack_promisor:1 ; +unsigned char sha1[20 ]; +struct revindex_entry *revindex; +/* something like ".git/objects/pack/xxxxx.pack" */ +char pack_name[FLEX_ARRAY ];/* more */ +}; +struct raw_object_store {/* * Path to the repository's object store.
@@ -59,10 +83,15 @@ struct raw_object_store {*/ char *objectdir; +struct packed_git *packed_git; +/* A most-recently-used ordered version of the packed_git list. */ +struct list_head packed_git_mru; +struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; }; -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL } + +#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL }void raw_object_store_clear (struct raw_object_store *o);
object.c +6 -0
@@ -466,4 +466,10 @@ void raw_object_store_clear(struct raw_object_store *o)free_alt_odbs (o); o->alt_odb_tail =NULL ; + +INIT_LIST_HEAD (&o->packed_git_mru); +/* +* TODO: call close_all_packs once migrated to +* take an object store argument +*/ }
pack-bitmap.c +2 -1
@@ -10,6 +10,7 @@#include "pack-revindex.h" #include "pack-objects.h" #include "packfile.h" +#include "repository.h" /* * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +336,7 @@ static int open_pack_bitmap(void)assert (!bitmap_git.map && !bitmap_git.loaded);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (open_pack_bitmap_1 (p) ==0 ) ret =0 ; }
packfile.c +20 -19
@@ -45,8 +45,6 @@ static unsigned int pack_open_fds;static unsigned int pack_max_fds;static size_t peak_pack_mapped;static size_t pack_mapped; -struct packed_git *packed_git; -LIST_HEAD (packed_git_mru);#define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; }
@@ -246,7 +244,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -316,7 +314,7 @@ void close_all_packs(void) {struct packed_git *p; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)if (p->do_not_close)die ("BUG: want to close pack marked 'do-not-close'" );else
@@ -384,7 +382,7 @@ static int close_one_pack(void)struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -686,8 +684,8 @@ void install_packed_git(struct packed_git *pack)if (pack->pack_fd !=-1 ) pack_open_fds++; - pack->next = packed_git; - packed_git = pack; + pack->next = the_repository->objects.packed_git; + the_repository->objects.packed_git = pack; }void (*report_garbage)(unsigned seen_bits,const char *path);
@@ -769,7 +767,8 @@ static void prepare_packed_git_one(char *objdir, int local) base_len = path.len;if (strip_suffix_mem (path.buf, &base_len,".idx" )) {/* Don't reopen a pack we already have. */ -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; + p = p->next) {size_t len; if (strip_suffix(p->pack_name, ".pack", &len) && len == base_len &&
@@ -820,7 +819,7 @@ unsigned long approximate_object_count(void)prepare_packed_git (); count =0 ; -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if (open_pack_index (p))continue ; count += p->num_objects;
@@ -869,18 +868,19 @@ static int sort_pack(const void *a_, const void *b_)static void rearrange_packed_git (void ) { - packed_git =llist_mergesort (packed_git, get_next_packed_git, - set_next_packed_git, sort_pack); + the_repository->objects.packed_git =llist_mergesort ( + the_repository->objects.packed_git, get_next_packed_git, + set_next_packed_git, sort_pack); }static void prepare_packed_git_mru (void ) {struct packed_git *p; -INIT_LIST_HEAD (&packed_git_mru); +INIT_LIST_HEAD (&the_repository->objects.packed_git_mru); -for (p = packed_git; p; p = p->next) -list_add_tail (&p->mru, &packed_git_mru); +for (p = the_repository->objects.packed_git; p; p = p->next) +list_add_tail (&p->mru, &the_repository->objects.packed_git_mru); }static int prepare_packed_git_run_once =0 ;
@@ -1014,7 +1014,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)struct packed_git *p;unsigned i; -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)for (i =0 ; i < p->num_bad_objects; i++)if (!hashcmp (sha1, p->bad_object_sha1 +20 * i))return p;
@@ -1845,13 +1845,14 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)struct list_head *pos;prepare_packed_git (); -if (!packed_git) +if (!the_repository->objects.packed_git)return 0 ; -list_for_each (pos, &packed_git_mru) { +list_for_each (pos, &the_repository->objects.packed_git_mru) { struct packed_git *p = list_entry(pos, struct packed_git, mru);if (fill_pack_entry (sha1, e, p)) { -list_move (&p->mru, &packed_git_mru); +list_move (&p->mru, + &the_repository->objects.packed_git_mru);return 1 ; } }
@@ -1898,7 +1899,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)int pack_errors =0 ;prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ; if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
repository.c +3 -1
@@ -7,7 +7,7 @@/* The main repository */ static struct repository the_repo = {NULL ,NULL , -RAW_OBJECT_STORE_INIT , +RAW_OBJECT_STORE_INIT (the_repo.objects),NULL ,NULL ,NULL ,NULL ,NULL ,NULL , &the_index,
@@ -143,6 +143,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree) repo->ignore_env =1 ; +INIT_LIST_HEAD (&repo->objects.packed_git_mru); +if (repo_init_gitdir (repo, gitdir)) goto error;
server-info.c +3 -2
@@ -1,4 +1,5 @@#include "cache.h" +#include "repository.h" #include "refs.h" #include "object.h" #include "commit.h"
@@ -200,7 +201,7 @@ static void init_pack_info(const char *infofile, int force) objdirlen =strlen (objdir);prepare_packed_git (); -for (p = packed_git; p; p = p->next) { +for (p = the_repository->objects.packed_git; p; p = p->next) {/* we ignore things on alternate path since they are * not available to the pullers in general. */
@@ -210,7 +211,7 @@ static void init_pack_info(const char *infofile, int force) } num_pack = i; info =xcalloc (num_pack,sizeof (struct pack_info *)); -for (i =0 , p = packed_git; p; p = p->next) { +for (i =0 , p = the_repository->objects.packed_git; p; p = p->next) {if (!p->pack_local)continue ; info[i] =xcalloc (1 ,sizeof (struct pack_info ));
sha1_name.c +3 -2
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)struct packed_git *p;prepare_packed_git (); -for (p = packed_git; p && !ds->ambiguous; p = p->next) +for (p = the_repository->objects.packed_git; p && !ds->ambiguous; + p = p->next)unique_in_pack (p, ds); }
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)struct packed_git *p;prepare_packed_git (); -for (p = packed_git; p; p = p->next) +for (p = the_repository->objects.packed_git; p; p = p->next)find_abbrev_len_for_pack (p, mad); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 6/27 object-store: close all packs upon clearing the object store
In reply to Stefan Beller--- builtin/am.c | 2 +- builtin/clone.c | 2 +- builtin/fetch.c | 2 +- builtin/merge.c | 2 +- builtin/receive-pack.c | 2 +- object.c | 7 +++---- packfile.c | 4 ++-- packfile.h | 2 +- 8 files changed, 11 insertions(+), 12 deletions(-)
builtin/am.c +1 -1
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)*/ if (!state->rebasing) {am_destroy (state); -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }
builtin/clone.c +1 -1
@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)transport_disconnect (transport);if (option_dissociate) { -close_all_packs (); +close_all_packs (&the_repository->objects);dissociate_from_references (); }
builtin/fetch.c +1 -1
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)string_list_clear (&list,0 ); -close_all_packs (); +close_all_packs (&the_repository->objects);argv_array_pushl (&argv_gc_auto,"gc" ,"--auto" ,NULL );if (verbosity <0 )
builtin/merge.c +1 -1
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,* We ignore errors in 'gc --auto', since the * user should see them. */ -close_all_packs (); +close_all_packs (&the_repository->objects);run_command_v_opt (argv_gc_auto,RUN_GIT_CMD ); } }
builtin/receive-pack.c +1 -1
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix) proc.git_cmd =1 ; proc.argv = argv_gc_auto; -close_all_packs (); +close_all_packs (&the_repository->objects);if (!start_command (&proc)) {if (use_sideband)copy_to_sideband (proc.err,-1 ,NULL );
object.c +3 -4
@@ -4,6 +4,7 @@#include "tree.h" #include "commit.h" #include "tag.h" +#include "packfile.h" static struct object **obj_hash;static int nr_objs, obj_hash_size;
@@ -468,8 +469,6 @@ void raw_object_store_clear(struct raw_object_store *o) o->alt_odb_tail =NULL ;INIT_LIST_HEAD (&o->packed_git_mru); -/* -* TODO: call close_all_packs once migrated to -* take an object store argument -*/ +close_all_packs (o); + o->packed_git =NULL ; }
packfile.c +2 -2
@@ -310,11 +310,11 @@ static void close_pack(struct packed_git *p)close_pack_index (p); } -void close_all_packs (void ) +void close_all_packs (struct raw_object_store *o) {struct packed_git *p; -for (p = the_repository->objects.packed_git; p; p = p->next) +for (p = o->packed_git; p; p = p->next)if (p->do_not_close)die ("BUG: want to close pack marked 'do-not-close'" );else
packfile.h +1 -1
@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void close_pack_windows (struct packed_git *); -extern void close_all_packs (void ); +extern void close_all_packs (struct raw_object_store *o);extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 7/27 pack: move prepare_packed_git_run_once to object store
In reply to Stefan BellerEach repository's object store can be initialized independently, so they must not share a run_once variable.
--- object-store.h | 8 +++++++- packfile.c | 7 +++---- 2 files changed, 10 insertions(+), 5 deletions(-)
object-store.h +7 -1
@@ -89,9 +89,15 @@ struct raw_object_store {struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; + +/* +* Whether packed_git has already been populated with this repository's +* packs. +*/ + unsigned packed_git_initialized : 1; }; -#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL } +#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0 }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +3 -4
@@ -883,12 +883,11 @@ static void prepare_packed_git_mru(void)list_add_tail (&p->mru, &the_repository->objects.packed_git_mru); } -static int prepare_packed_git_run_once =0 ;void prepare_packed_git (void ) {struct alternate_object_database *alt; -if (prepare_packed_git_run_once) +if (the_repository->objects.packed_git_initialized)return ;prepare_packed_git_one (get_object_directory (),1 );prepare_alt_odb ();
@@ -896,13 +895,13 @@ void prepare_packed_git(void)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();prepare_packed_git_mru (); - prepare_packed_git_run_once =1 ; + the_repository->objects.packed_git_initialized =1 ; }void reprepare_packed_git (void ) { approximate_object_count_valid =0 ; - prepare_packed_git_run_once =0 ; + the_repository->objects.packed_git_initialized =0 ;prepare_packed_git (); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 8/27 pack: move approximate object count to object store
In reply to Stefan BellerThe approximate_object_count() function maintains a rough count of objects in a repository to estimate how long object name abbreviates should be. Object names are scoped to a repository and the appropriate length may differ by repository, so the object count should not be global.
--- object-store.h | 10 +++++++++- packfile.c | 11 +++++------ 2 files changed, 14 insertions(+), 7 deletions(-)
object-store.h +9 -1
@@ -90,6 +90,14 @@ struct raw_object_store {struct alternate_object_database *alt_odb_list;struct alternate_object_database **alt_odb_tail; +/* +* A fast, rough count of the number of objects in the repository. +* These two fields are not meant for direct access. Use +* approximate_object_count() instead. +*/ +unsigned long approximate_object_count; + unsigned approximate_object_count_valid : 1; +/* * Whether packed_git has already been populated with this repository's * packs.
@@ -97,7 +105,7 @@ struct raw_object_store { unsigned packed_git_initialized : 1; }; -#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0 } +#define RAW_OBJECT_STORE_INIT (o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }void raw_object_store_clear (struct raw_object_store *o);
packfile.c +5 -6
@@ -802,8 +802,6 @@ static void prepare_packed_git_one(char *objdir, int local)strbuf_release (&path); } -static int approximate_object_count_valid; -/* * Give a fast, rough count of the number of objects in the repository. This * ignores loose objects completely. If you have a lot of them, then either
@@ -813,8 +811,8 @@ static int approximate_object_count_valid;*/ unsigned long approximate_object_count (void ) { -static unsigned long count; -if (!approximate_object_count_valid) { +if (!the_repository->objects.approximate_object_count_valid) { +unsigned long count;struct packed_git *p;prepare_packed_git ();
@@ -824,8 +822,9 @@ unsigned long approximate_object_count(void)continue ; count += p->num_objects; } + the_repository->objects.approximate_object_count = count; } -return count; +return the_repository->objects.approximate_object_count; }static void *get_next_packed_git (const void *p)
@@ -900,7 +899,7 @@ void prepare_packed_git(void)void reprepare_packed_git (void ) { - approximate_object_count_valid =0 ; + the_repository->objects.approximate_object_count_valid =0 ; the_repository->objects.packed_git_initialized =0 ;prepare_packed_git (); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 9/27 sha1_file: add raw_object_store argument to alt_odb_usable
In reply to Stefan BellerAdd a raw_object_store to alt_odb_usable to be more specific about which repository to act on. The choice of the repository is delegated to its only caller link_alt_odb_entry.
--- sha1_file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
sha1_file.c +5 -3
@@ -344,7 +344,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,/* * Return non-zero iff the path is usable as an alternate object database. */ -static int alt_odb_usable (struct strbuf *path,const char *normalized_objdir) +static int alt_odb_usable (struct raw_object_store *o, +struct strbuf *path, +const char *normalized_objdir) {struct alternate_object_database *alt;
@@ -360,7 +362,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)* Prevent the common mistake of listing the same * thing twice, or object directory itself. */ -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +for (alt = o->alt_odb_list; alt; alt = alt->next) {if (!fspathcmp (path->buf, alt->path))return 0 ; }
@@ -412,7 +414,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,while (pathbuf.len && pathbuf.buf[pathbuf.len -1 ] =='/' )strbuf_setlen (&pathbuf, pathbuf.len -1 ); -if (!alt_odb_usable (&pathbuf, normalized_objdir)) { +if (!alt_odb_usable (&the_repository->objects, &pathbuf, normalized_objdir)) {strbuf_release (&pathbuf);return -1 ; }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 10/27 sha1_file: add repository argument to link_alt_odb_entry
In reply to Stefan BellerAdd a repository argument to allow the link_alt_odb_entry caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
Since the implementation does not yet work with other repositories, use a wrapper macro to enforce that the caller passes in the_repository as the first argument. It would be more appealing to use BUILD_ASSERT_OR_ZERO to enforce this, but that doesn't work because it requires a compile-time constant and common compilers like gcc 4.8.4 do not consider "r == the_repository" a compile-time constant.
This and the following three patches add repository arguments to link_alt_odb_entry, read_info_alternates, link_alt_odb_entries and prepare_alt_odb. Three out of the four functions are found in a recursive call chain, calling each other, and one of them accesses the repositories objectdir (which was migrated; it was an obvious choice) and ignore_env (which we need to keep in the repository struct for clarify); hence we will pass through the repository unlike just the object store object + the ignore_env flag.
--- sha1_file.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
sha1_file.c +5 -3
@@ -388,8 +388,9 @@ static int alt_odb_usable(struct raw_object_store *o,* terminating NUL. */ static void read_info_alternates (const char * relative_base,int depth); -static int link_alt_odb_entry (const char *entry,const char *relative_base, -int depth,const char *normalized_objdir) +#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n) +static int link_alt_odb_entry_the_repository (const char *entry, +const char *relative_base,int depth,const char *normalized_objdir) {struct alternate_object_database *ent;struct strbuf pathbuf =STRBUF_INIT ;
@@ -486,7 +487,8 @@ static void link_alt_odb_entries(const char *alt, int sep, alt =parse_alt_odb_entry (alt, sep, &entry);if (!entry.len)continue ; -link_alt_odb_entry (entry.buf, relative_base, depth, objdirbuf.buf); +link_alt_odb_entry (the_repository, entry.buf, + relative_base, depth, objdirbuf.buf); }strbuf_release (&entry);strbuf_release (&objdirbuf);
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 11/27 sha1_file: add repository argument to read_info_alternates
In reply to Stefan BellerSee previous patch for explanation.
--- sha1_file.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-)
sha1_file.c +7 -4
@@ -387,7 +387,9 @@ static int alt_odb_usable(struct raw_object_store *o,* SHA1, an extra slash for the first level indirection, and the * terminating NUL. */ -static void read_info_alternates (const char * relative_base,int depth); +#define read_info_alternates (r, rb, d) read_info_alternates_##r(rb, d) +static void read_info_alternates_the_repository (const char *relative_base, +int depth);#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)static int link_alt_odb_entry_the_repository (const char *entry,const char *relative_base,int depth,const char *normalized_objdir)
@@ -428,7 +430,7 @@ static int link_alt_odb_entry_the_repository(const char *entry, ent->next =NULL ;/* recursively add alternates */ -read_info_alternates (pathbuf.buf, depth +1 ); +read_info_alternates (the_repository, pathbuf.buf, depth +1 );strbuf_release (&pathbuf);return 0 ;
@@ -494,7 +496,8 @@ static void link_alt_odb_entries(const char *alt, int sep,strbuf_release (&objdirbuf); } -static void read_info_alternates (const char * relative_base,int depth) +static void read_info_alternates_the_repository (const char *relative_base, +int depth) {char *path;struct strbuf buf =STRBUF_INIT ;
@@ -678,7 +681,7 @@ void prepare_alt_odb(void) &the_repository->objects.alt_odb_list;link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 ); -read_info_alternates (get_object_directory (),0 ); +read_info_alternates (the_repository,get_object_directory (),0 ); }/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 12/27 sha1_file: add repository argument to link_alt_odb_entries
In reply to Stefan BellerSee previous patch for explanation.
--- sha1_file.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
sha1_file.c +13 -6
@@ -465,8 +465,12 @@ static const char *parse_alt_odb_entry(const char *string,return end; } -static void link_alt_odb_entries (const char *alt,int sep, -const char *relative_base,int depth) +#define link_alt_odb_entries (r, a, s, rb, d) \ + link_alt_odb_entries_##r(a, s, rb, d) +static void link_alt_odb_entries_the_repository (const char *alt, +int sep, +const char *relative_base, +int depth) {struct strbuf objdirbuf =STRBUF_INIT ;struct strbuf entry =STRBUF_INIT ;
@@ -509,7 +513,7 @@ static void read_info_alternates_the_repository(const char *relative_base,return ; } -link_alt_odb_entries (buf.buf,'\n' , relative_base, depth); +link_alt_odb_entries (the_repository, buf.buf,'\n' , relative_base, depth);strbuf_release (&buf);free (path); }
@@ -563,7 +567,8 @@ void add_to_alternates_file(const char *reference)if (commit_lock_file (&lock))die_errno ("unable to move new alternates file into place" );if (the_repository->objects.alt_odb_tail) -link_alt_odb_entries (reference,'\n' ,NULL ,0 ); +link_alt_odb_entries (the_repository, reference, +'\n' ,NULL ,0 ); }free (alts); }
@@ -576,7 +581,8 @@ void add_to_alternates_memory(const char *reference)*/ prepare_alt_odb (); -link_alt_odb_entries (reference,'\n' ,NULL ,0 ); +link_alt_odb_entries (the_repository, reference, +'\n' ,NULL ,0 ); }/*
@@ -679,7 +685,8 @@ void prepare_alt_odb(void) the_repository->objects.alt_odb_tail = &the_repository->objects.alt_odb_list; -link_alt_odb_entries (alt,PATH_SEP ,NULL ,0 ); +link_alt_odb_entries (the_repository, alt, +PATH_SEP ,NULL ,0 );read_info_alternates (the_repository,get_object_directory (),0 ); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 13/27 sha1_file: add repository argument to prepare_alt_odb
In reply to Stefan BellerSee previous patch for explanation.
--- builtin/fsck.c | 2 +- object-store.h | 3 ++- packfile.c | 2 +- sha1_file.c | 13 +++++++------ sha1_name.c | 2 +- 5 files changed, 12 insertions(+), 10 deletions(-)
builtin/fsck.c +1 -1
@@ -722,7 +722,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix) }else {fsck_object_dir (get_object_directory ()); -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)fsck_object_dir (alt->path);
object-store.h +2 -1
@@ -20,7 +20,8 @@ struct alternate_object_database {char path[FLEX_ARRAY ]; }; -void prepare_alt_odb (void ); +#define prepare_alt_odb (r) prepare_alt_odb_##r() +void prepare_alt_odb_the_repository (void );char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);int foreach_alt_odb (alt_odb_fn ,void *);
packfile.c +1 -1
@@ -889,7 +889,7 @@ void prepare_packed_git(void)if (the_repository->objects.packed_git_initialized)return ;prepare_packed_git_one (get_object_directory (),1 ); -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)prepare_packed_git_one (alt->path,0 );rearrange_packed_git ();
sha1_file.c +7 -6
@@ -23,6 +23,7 @@#include "sha1-lookup.h" #include "bulk-checkin.h" #include "repository.h" +#include "object-store.h" #include "streaming.h" #include "dir.h" #include "list.h"
@@ -579,7 +580,7 @@ void add_to_alternates_memory(const char *reference)* Make sure alternates are initialized, or else our entry may be * overwritten when they are. */ -prepare_alt_odb (); +prepare_alt_odb (the_repository);link_alt_odb_entries (the_repository, reference,'\n' ,NULL ,0 );
@@ -665,7 +666,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)struct alternate_object_database *ent;int r =0 ; -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) { r =fn (ent, cb);if (r)
@@ -674,7 +675,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb (void ) +void prepare_alt_odb_the_repository (void ) {const char *alt;
@@ -728,7 +729,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)static int check_and_freshen_nonlocal (const unsigned char *sha1,int freshen) {struct alternate_object_database *alt; -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {const char *path =alt_sha1_path (alt, sha1);if (check_and_freshen_file (path, freshen))
@@ -887,7 +888,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,if (!lstat (*path, st))return 0 ; -prepare_alt_odb (); +prepare_alt_odb (the_repository); errno =ENOENT ;for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);
@@ -918,7 +919,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)return fd; most_interesting_errno = errno; -prepare_alt_odb (); +prepare_alt_odb (the_repository);for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);
sha1_name.c +1 -1
@@ -353,7 +353,7 @@ static int init_object_disambiguation(const char *name, int len, ds->len = len; ds->hex_pfx[len] ='\0' ; -prepare_alt_odb (); +prepare_alt_odb (the_repository);return 0 ; }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 14/27 sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
In reply to Stefan BellerActually this also allows read_info_alternates and link_alt_odb_entry to handle arbitrary repositories, but link_alt_odb_entries is the most interesting function in this set of functions, hence the commit subject.
These functions span a strongly connected component in the function graph, i.e. the recursive call chain might look like
-> link_alt_odb_entries
-> link_alt_odb_entry
-> read_info_alternates
-> link_alt_odb_entriesThat is why we need to convert all these functions at the same time.
--- object-store.h | 4 ++++ sha1_file.c | 36 ++++++++++++++++-------------------- 2 files changed, 20 insertions(+), 20 deletions(-)
object-store.h +4 -0
@@ -18,6 +18,10 @@ struct alternate_object_database {char loose_objects_subdir_seen[256 ];struct oid_array loose_objects_cache; +/* +* Path to the alternative object store. If this is a relative path, +* it is relative to the current working directory. +*/ char path[FLEX_ARRAY ]; };#define prepare_alt_odb (r) prepare_alt_odb_##r()
sha1_file.c +16 -20
@@ -388,11 +388,10 @@ static int alt_odb_usable(struct raw_object_store *o,* SHA1, an extra slash for the first level indirection, and the * terminating NUL. */ -#define read_info_alternates (r, rb, d) read_info_alternates_##r(rb, d) -static void read_info_alternates_the_repository (const char *relative_base, -int depth); -#define link_alt_odb_entry (r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n) -static int link_alt_odb_entry_the_repository (const char *entry, +static void read_info_alternates (struct repository *r, +const char *relative_base, +int depth); +static int link_alt_odb_entry (struct repository *r,const char *entry,const char *relative_base,int depth,const char *normalized_objdir) {struct alternate_object_database *ent;
@@ -418,7 +417,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,while (pathbuf.len && pathbuf.buf[pathbuf.len -1 ] =='/' )strbuf_setlen (&pathbuf, pathbuf.len -1 ); -if (!alt_odb_usable (&the_repository->objects, &pathbuf, normalized_objdir)) { +if (!alt_odb_usable (&r->objects, &pathbuf, normalized_objdir)) {strbuf_release (&pathbuf);return -1 ; }
@@ -426,12 +425,12 @@ static int link_alt_odb_entry_the_repository(const char *entry, ent =alloc_alt_odb (pathbuf.buf);/* add the alternate entry */ - *the_repository->objects.alt_odb_tail = ent; - the_repository->objects.alt_odb_tail = &(ent->next); + *r->objects.alt_odb_tail = ent; + r->objects.alt_odb_tail = &(ent->next); ent->next =NULL ;/* recursively add alternates */ -read_info_alternates (the_repository, pathbuf.buf, depth +1 ); +read_info_alternates (r, pathbuf.buf, depth +1 );strbuf_release (&pathbuf);return 0 ;
@@ -466,12 +465,8 @@ static const char *parse_alt_odb_entry(const char *string,return end; } -#define link_alt_odb_entries (r, a, s, rb, d) \ - link_alt_odb_entries_##r(a, s, rb, d) -static void link_alt_odb_entries_the_repository (const char *alt, -int sep, -const char *relative_base, -int depth) +static void link_alt_odb_entries (struct repository *r,const char *alt, +int sep,const char *relative_base,int depth) {struct strbuf objdirbuf =STRBUF_INIT ;struct strbuf entry =STRBUF_INIT ;
@@ -485,7 +480,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,return ; } -strbuf_add_absolute_path (&objdirbuf,get_object_directory ()); +strbuf_add_absolute_path (&objdirbuf, r->objects.objectdir);if (strbuf_normalize_path (&objdirbuf) <0 )die ("unable to normalize object directory: %s" , objdirbuf.buf);
@@ -494,15 +489,16 @@ static void link_alt_odb_entries_the_repository(const char *alt, alt =parse_alt_odb_entry (alt, sep, &entry);if (!entry.len)continue ; -link_alt_odb_entry (the_repository, entry.buf, +link_alt_odb_entry (r, entry.buf, relative_base, depth, objdirbuf.buf); }strbuf_release (&entry);strbuf_release (&objdirbuf); } -static void read_info_alternates_the_repository (const char *relative_base, -int depth) +static void read_info_alternates (struct repository *r, +const char *relative_base, +int depth) {char *path;struct strbuf buf =STRBUF_INIT ;
@@ -514,7 +510,7 @@ static void read_info_alternates_the_repository(const char *relative_base,return ; } -link_alt_odb_entries (the_repository, buf.buf,'\n' , relative_base, depth); +link_alt_odb_entries (r, buf.buf,'\n' , relative_base, depth);strbuf_release (&buf);free (path); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 15/27 sha1_file: allow prepare_alt_odb to handle arbitrary repositories
In reply to Stefan Beller--- object-store.h | 3 +-- sha1_file.c | 12 +++++------- 2 files changed, 6 insertions(+), 9 deletions(-)
object-store.h +1 -2
@@ -24,8 +24,7 @@ struct alternate_object_database {*/ char path[FLEX_ARRAY ]; }; -#define prepare_alt_odb (r) prepare_alt_odb_##r() -void prepare_alt_odb_the_repository (void ); +void prepare_alt_odb (struct repository *r);char *compute_alternate_path (const char *path,struct strbuf *err);typedef int alt_odb_fn (struct alternate_object_database *,void *);int foreach_alt_odb (alt_odb_fn ,void *);
sha1_file.c +5 -7
@@ -671,21 +671,19 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)return r; } -void prepare_alt_odb_the_repository (void ) +void prepare_alt_odb (struct repository *r) {const char *alt; -if (the_repository->objects.alt_odb_tail) +if (r->objects.alt_odb_tail)return ; alt =getenv (ALTERNATE_DB_ENVIRONMENT ); - the_repository->objects.alt_odb_tail = - &the_repository->objects.alt_odb_list; -link_alt_odb_entries (the_repository, alt, -PATH_SEP ,NULL ,0 ); + r->objects.alt_odb_tail = &r->objects.alt_odb_list; +link_alt_odb_entries (r, alt,PATH_SEP ,NULL ,0 ); -read_info_alternates (the_repository,get_object_directory (),0 ); +read_info_alternates (r, r->objects.objectdir,0 ); }/* Returns 1 if we have successfully freshened the file, 0 otherwise. */
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 16/27 sha1_file: add repository argument to sha1_file_name
In reply to Stefan BellerAdd a repository argument to allow sha1_file_name callers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
While at it, move the declaration to object-store.h, where it should be easier to find.
--- cache.h | 6 ------ http-walker.c | 3 ++- http.c | 5 ++--- object-store.h | 7 +++++++ sha1_file.c | 10 +++++----- 5 files changed, 16 insertions(+), 15 deletions(-)
cache.h +0 -6
@@ -936,12 +936,6 @@ extern void check_repository_format(void);#define DATA_CHANGED 0x0020#define TYPE_CHANGED 0x0040 -/* -* Put in `buf` the name of the file in the local object database that -* would be used to store a loose object with the specified sha1. -*/ -extern void sha1_file_name (struct strbuf *buf,const unsigned char *sha1); -/* * Return an abbreviated sha1 unique within this repository's object database. * The result will be at least `len` characters long, and will be NUL
http-walker.c +2 -1
@@ -1,4 +1,5 @@#include "cache.h" +#include "repository.h" #include "commit.h" #include "walker.h" #include "http.h"
@@ -545,7 +546,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1) ret =error ("File %s has bad hash" , hex); }else if (req->rename <0 ) {struct strbuf buf =STRBUF_INIT ; -sha1_file_name (&buf, req->sha1); +sha1_file_name (the_repository, &buf, req->sha1); ret =error ("unable to write sha1 filename %s" , buf.buf);strbuf_release (&buf); }
http.c +2 -3
@@ -2246,7 +2246,7 @@ struct http_object_request *new_http_object_request(const char *base_url,hashcpy (freq->sha1, sha1); freq->localfile =-1 ; -sha1_file_name (&filename, sha1); +sha1_file_name (the_repository, &filename, sha1);snprintf (freq->tmpfile,sizeof (freq->tmpfile),"%s.temp" , filename.buf);
@@ -2395,8 +2395,7 @@ int finish_http_object_request(struct http_object_request *freq)unlink_or_warn (freq->tmpfile);return -1 ; } - -sha1_file_name (&filename, freq->sha1); +sha1_file_name (the_repository, &filename, freq->sha1); freq->rename =finalize_object_file (freq->tmpfile, filename.buf);strbuf_release (&filename);
object-store.h +7 -0
@@ -113,4 +113,11 @@ struct raw_object_store {void raw_object_store_clear (struct raw_object_store *o); +/* +* Put in `buf` the name of the file in the local object database that +* would be used to store a loose object with the specified sha1. +*/ +#define sha1_file_name (r, b, s) sha1_file_name_##r(b, s) +void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1); +#endif /* OBJECT_STORE_H */
sha1_file.c +5 -5
@@ -321,7 +321,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1) } } -void sha1_file_name (struct strbuf *buf,const unsigned char *sha1) +void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1) {strbuf_addstr (buf,get_object_directory ());strbuf_addch (buf,'/' );
@@ -715,7 +715,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&buf, sha1); +sha1_file_name (the_repository, &buf, sha1);return check_and_freshen_file (buf.buf, freshen); }
@@ -876,7 +876,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&buf, sha1); +sha1_file_name (the_repository, &buf, sha1); *path = buf.buf;if (!lstat (*path, st))
@@ -905,7 +905,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (&buf, sha1); +sha1_file_name (the_repository, &buf, sha1); *path = buf.buf; fd =git_open (*path);
@@ -1591,7 +1591,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,static struct strbuf filename =STRBUF_INIT ;strbuf_reset (&filename); -sha1_file_name (&filename, oid->hash); +sha1_file_name (the_repository, &filename, oid->hash); fd =create_tmpfile (&tmp_file, filename.buf);if (fd <0 ) {
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 17/27 sha1_file: add repository argument to stat_sha1_file
In reply to Stefan BellerAdd a repository argument to allow the stat_sha1_file caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
sha1_file.c +5 -4
@@ -869,8 +869,9 @@ int git_open_cloexec(const char *name, int flags)* Note that it may point to static storage and is only valid until another * call to sha1_file_name(), etc. */ -static int stat_sha1_file (const unsigned char *sha1,struct stat *st, -const char **path) +#define stat_sha1_file (r, s, st, p) stat_sha1_file_##r(s, st, p) +static int stat_sha1_file_the_repository (const unsigned char *sha1, +struct stat *st,const char **path) {struct alternate_object_database *alt;static struct strbuf buf =STRBUF_INIT ;
@@ -1176,7 +1177,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {const char *path;struct stat st; -if (stat_sha1_file (sha1, &st, &path) <0 ) +if (stat_sha1_file (the_repository, sha1, &st, &path) <0 )return -1 ;if (oi->disk_sizep) *oi->disk_sizep = st.st_size;
@@ -1390,7 +1391,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,die ("replacement %s not found for %s" ,sha1_to_hex (repl),sha1_to_hex (sha1)); -if (!stat_sha1_file (repl, &st, &path)) +if (!stat_sha1_file (the_repository, repl, &st, &path))die ("loose object %s (stored in %s) is corrupt" ,sha1_to_hex (repl), path);
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 18/27 sha1_file: add repository argument to open_sha1_file
In reply to Stefan BellerAdd a repository argument to allow the open_sha1_file caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
sha1_file.c +4 -2
@@ -898,7 +898,9 @@ static int stat_sha1_file_the_repository(const unsigned char *sha1,* Like stat_sha1_file(), but actually open the object and return the * descriptor. See the caveats on the "path" parameter above. */ -static int open_sha1_file (const unsigned char *sha1,const char **path) +#define open_sha1_file (r, s, p) open_sha1_file_##r(s, p) +static int open_sha1_file_the_repository (const unsigned char *sha1, +const char **path) {int fd;struct alternate_object_database *alt;
@@ -941,7 +943,7 @@ static void *map_sha1_file_1(const char *path,if (path) fd =git_open (path);else - fd =open_sha1_file (sha1, &path); + fd =open_sha1_file (the_repository, sha1, &path); map =NULL ;if (fd >=0 ) {struct stat st;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 19/27 sha1_file: add repository argument to map_sha1_file_1
In reply to Stefan BellerAdd a repository argument to allow the map_sha1_file_1 caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-)
sha1_file.c +6 -5
@@ -933,9 +933,10 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,* Map the loose object at "path" if it is not NULL, or the path found by * searching for a loose object named "sha1". */ -static void *map_sha1_file_1 (const char *path, -const unsigned char *sha1, -unsigned long *size) +#define map_sha1_file_1 (r, p, s, si) map_sha1_file_1_##r(p, s, si) +static void *map_sha1_file_1_the_repository (const char *path, +const unsigned char *sha1, +unsigned long *size) {void *map;int fd;
@@ -964,7 +965,7 @@ static void *map_sha1_file_1(const char *path,void *map_sha1_file (const unsigned char *sha1,unsigned long *size) { -return map_sha1_file_1 (NULL , sha1, size); +return map_sha1_file_1 (the_repository,NULL , sha1, size); } static int unpack_sha1_short_header(git_zstream *stream,
@@ -2200,7 +2201,7 @@ int read_loose_object(const char *path, *contents =NULL ; - map =map_sha1_file_1 (path,NULL , &mapsize); + map =map_sha1_file_1 (the_repository, path,NULL , &mapsize);if (!map) {error_errno ("unable to mmap %s" , path); goto out;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 20/27 sha1_file: add repository argument to map_sha1_file
In reply to Stefan BellerAdd a repository argument to allow map_sha1_file callers to be more specific about which repository to handle. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
While at it, move the declaration to object-store.h, where it should be easier to find.
--- cache.h | 1 - object-store.h | 3 +++ sha1_file.c | 4 ++-- streaming.c | 5 ++++- 4 files changed, 9 insertions(+), 4 deletions(-)
cache.h +0 -1
@@ -1226,7 +1226,6 @@ extern int force_object_loose(const struct object_id *oid, time_t mtime);extern int git_open_cloexec (const char *name,int flags);#define git_open (name) git_open_cloexec(name, O_RDONLY) -extern void *map_sha1_file (const unsigned char *sha1,unsigned long *size);extern int unpack_sha1_header (git_zstream *stream,unsigned char *map,unsigned long mapsize,void *buffer,unsigned long bufsiz);extern int parse_sha1_header (const char *hdr,unsigned long *sizep);
object-store.h +3 -0
@@ -120,4 +120,7 @@ void raw_object_store_clear(struct raw_object_store *o);#define sha1_file_name (r, b, s) sha1_file_name_##r(b, s)void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1); +#define map_sha1_file (r, s, sz) map_sha1_file_##r(s, sz) +void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size); +#endif /* OBJECT_STORE_H */
sha1_file.c +2 -2
@@ -963,7 +963,7 @@ static void *map_sha1_file_1_the_repository(const char *path,return map; } -void *map_sha1_file (const unsigned char *sha1,unsigned long *size) +void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size) {return map_sha1_file_1 (the_repository,NULL , sha1, size); }
@@ -1187,7 +1187,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,return 0 ; } - map =map_sha1_file (sha1, &mapsize); + map =map_sha1_file (the_repository, sha1, &mapsize);if (!map)return -1 ;
streaming.c +4 -1
@@ -3,6 +3,8 @@*/ #include "cache.h" #include "streaming.h" +#include "repository.h" +#include "object-store.h" #include "packfile.h" enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {static open_method_decl (loose) { - st->u.loose.mapped =map_sha1_file (sha1, &st->u.loose.mapsize); + st->u.loose.mapped =map_sha1_file (the_repository, + sha1, &st->u.loose.mapsize);if (!st->u.loose.mapped)return -1 ; if ((unpack_sha1_header(&st->z,
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 21/27 sha1_file: add repository argument to sha1_loose_object_info
In reply to Stefan BellerAdd a repository argument to allow the sha1_loose_object_info caller to be more specific about which repository to act on. This is a small mechanical change; it doesn't change the implementation to handle repositories other than the_repository yet.
As with the previous commits, use a macro to catch callers passing a repository other than the_repository at compile time.
--- sha1_file.c | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-)
sha1_file.c +5 -4
@@ -1154,9 +1154,10 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -static int sha1_loose_object_info (const unsigned char *sha1, -struct object_info *oi, -int flags) +#define sha1_loose_object_info (r, s, o, f) sha1_loose_object_info_##r(s, o, f) +static int sha1_loose_object_info_the_repository (const unsigned char *sha1, +struct object_info *oi, +int flags) {int status =0 ;unsigned long mapsize;
@@ -1275,7 +1276,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,break ;/* Most likely it's a loose object. */ -if (!sha1_loose_object_info (real, oi, flags)) +if (!sha1_loose_object_info (the_repository, real, oi, flags))return 0 ;/* Not a loose object; someone else may have just packed it. */
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 22/27 sha1_file: allow sha1_file_name to handle arbitrary repositories
In reply to Stefan Beller--- object-store.h | 3 +-- sha1_file.c | 4 ++-- 2 files changed, 3 insertions(+), 4 deletions(-)
object-store.h +1 -2
@@ -117,8 +117,7 @@ void raw_object_store_clear(struct raw_object_store *o);* Put in `buf` the name of the file in the local object database that * would be used to store a loose object with the specified sha1. */ -#define sha1_file_name (r, b, s) sha1_file_name_##r(b, s) -void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1); +void sha1_file_name (struct repository *r,struct strbuf *buf,const unsigned char *sha1);#define map_sha1_file (r, s, sz) map_sha1_file_##r(s, sz)void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size);
sha1_file.c +2 -2
@@ -321,9 +321,9 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1) } } -void sha1_file_name_the_repository (struct strbuf *buf,const unsigned char *sha1) +void sha1_file_name (struct repository *r,struct strbuf *buf,const unsigned char *sha1) { -strbuf_addstr (buf,get_object_directory ()); +strbuf_addstr (buf, r->objects.objectdir);strbuf_addch (buf,'/' );fill_sha1_path (buf, sha1); }
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 23/27 sha1_file: allow stat_sha1_file to handle arbitrary repositories
In reply to Stefan Beller--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -869,23 +869,22 @@ int git_open_cloexec(const char *name, int flags)* Note that it may point to static storage and is only valid until another * call to sha1_file_name(), etc. */ -#define stat_sha1_file (r, s, st, p) stat_sha1_file_##r(s, st, p) -static int stat_sha1_file_the_repository (const unsigned char *sha1, -struct stat *st,const char **path) +static int stat_sha1_file (struct repository *r,const unsigned char *sha1, +struct stat *st,const char **path) {struct alternate_object_database *alt;static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (the_repository, &buf, sha1); +sha1_file_name (r, &buf, sha1); *path = buf.buf;if (!lstat (*path, st))return 0 ; -prepare_alt_odb (the_repository); +prepare_alt_odb (r); errno =ENOENT ; -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +for (alt = r->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1);if (!lstat (*path, st))return 0 ;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 24/27 sha1_file: allow open_sha1_file to handle arbitrary repositories
In reply to Stefan Beller--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -897,9 +897,8 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,* Like stat_sha1_file(), but actually open the object and return the * descriptor. See the caveats on the "path" parameter above. */ -#define open_sha1_file (r, s, p) open_sha1_file_##r(s, p) -static int open_sha1_file_the_repository (const unsigned char *sha1, -const char **path) +static int open_sha1_file (struct repository *r, +const unsigned char *sha1,const char **path) {int fd;struct alternate_object_database *alt;
@@ -907,7 +906,7 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,static struct strbuf buf =STRBUF_INIT ;strbuf_reset (&buf); -sha1_file_name (the_repository, &buf, sha1); +sha1_file_name (r, &buf, sha1); *path = buf.buf; fd =git_open (*path);
@@ -915,8 +914,8 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,return fd; most_interesting_errno = errno; -prepare_alt_odb (the_repository); -for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) { +prepare_alt_odb (r); +for (alt = r->objects.alt_odb_list; alt; alt = alt->next) { *path =alt_sha1_path (alt, sha1); fd =git_open (*path);if (fd >=0 )
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 25/27 sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
--- sha1_file.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-)
sha1_file.c +3 -5
@@ -931,10 +931,8 @@ static int open_sha1_file(struct repository *r,* Map the loose object at "path" if it is not NULL, or the path found by * searching for a loose object named "sha1". */ -#define map_sha1_file_1 (r, p, s, si) map_sha1_file_1_##r(p, s, si) -static void *map_sha1_file_1_the_repository (const char *path, -const unsigned char *sha1, -unsigned long *size) +static void *map_sha1_file_1 (struct repository *r,const char *path, +const unsigned char *sha1,unsigned long *size) {void *map;int fd;
@@ -942,7 +940,7 @@ static void *map_sha1_file_1_the_repository(const char *path,if (path) fd =git_open (path);else - fd =open_sha1_file (the_repository, sha1, &path); + fd =open_sha1_file (r, sha1, &path); map =NULL ;if (fd >=0 ) {struct stat st;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 26/27 sha1_file: allow map_sha1_file to handle arbitrary repositories
In reply to Stefan Beller--- object-store.h | 3 +-- sha1_file.c | 5 +++-- 2 files changed, 4 insertions(+), 4 deletions(-)
object-store.h +1 -2
@@ -119,7 +119,6 @@ void raw_object_store_clear(struct raw_object_store *o);*/ void sha1_file_name (struct repository *r,struct strbuf *buf,const unsigned char *sha1); -#define map_sha1_file (r, s, sz) map_sha1_file_##r(s, sz) -void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size); +void *map_sha1_file (struct repository *r,const unsigned char *sha1,unsigned long *size);#endif /* OBJECT_STORE_H */
sha1_file.c +3 -2
@@ -959,9 +959,10 @@ static void *map_sha1_file_1(struct repository *r, const char *path,return map; } -void *map_sha1_file_the_repository (const unsigned char *sha1,unsigned long *size) +void *map_sha1_file (struct repository *r, +const unsigned char *sha1,unsigned long *size) { -return map_sha1_file_1 (the_repository,NULL , sha1, size); +return map_sha1_file_1 (r,NULL , sha1, size); } static int unpack_sha1_short_header(git_zstream *stream,
Signature · 1 line
2.16.1.291.g4437f3f132-goog
PATCH v4 27/27 sha1_file: allow sha1_loose_object_info to handle arbitrary repositories
In reply to Stefan BellerFrom: Jonathan Nieder <[email protected]>
--- sha1_file.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
sha1_file.c +5 -6
@@ -1151,10 +1151,9 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -#define sha1_loose_object_info (r, s, o, f) sha1_loose_object_info_##r(s, o, f) -static int sha1_loose_object_info_the_repository (const unsigned char *sha1, -struct object_info *oi, -int flags) +static int sha1_loose_object_info (struct repository *r, +const unsigned char *sha1, +struct object_info *oi,int flags) {int status =0 ;unsigned long mapsize;
@@ -1178,14 +1177,14 @@ static int sha1_loose_object_info_the_repository(const unsigned char *sha1,if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {const char *path;struct stat st; -if (stat_sha1_file (the_repository, sha1, &st, &path) <0 ) +if (stat_sha1_file (r, sha1, &st, &path) <0 )return -1 ;if (oi->disk_sizep) *oi->disk_sizep = st.st_size;return 0 ; } - map =map_sha1_file (the_repository, sha1, &mapsize); + map =map_sha1_file (r, sha1, &mapsize);if (!map)return -1 ;
Signature · 1 line
2.16.1.291.g4437f3f132-goog
I notice that there are a few global state (or configuration rather) left after this: packed_git_window_size, packed_git_limit and delta_base_cache_limit. These can be changed by $GIT_DIR/config, but since it's still global, a submodule repository will use the same settings of its super repository. If $SUBMODULE/config changes any of these, they are ignored.
The natural thing to do is move these to raw_object_store too (and repo_submodule_init needs to check submodule's config file). But one may argue these should be per-process instead of per-repo though. I don't know. But I thought I should mention this.
Signature · 1 line
Duy
Quoted 6 lines · Duy Nguyen
I notice that there are a few global state (or configuration rather) left after this:
packed_git_window_size,packed_git_limitanddelta_base_cache_limit. These can be changed by $GIT_DIR/config, but since it's still global, a submodule repository will use the same settings of its super repository. If $SUBMODULE/config changes any of these, they are ignored.
That sounds all packing related, which I plan on working on next.
Quoted 4 lines
The natural thing to do is move these to
raw_object_storetoo (andrepo_submodule_initneeds to check submodule's config file). But one may argue these should be per-process instead of per-repo though. I don't know. But I thought I should mention this.
For now a process and a repository is the same as git-gc or git-repack doesn't know about the --recurse-submodules flag, yet.
I wonder if we ever want to teach those commands the submodule recursion, because of the issue you bring up, which settings do we apply for a submodule? Traditionally we'd just have the command line override the configuration, which I don't know if it is a good idea for these settings.
Thanks,
Stefan
Quoted 6 lines · Stefan Beller
Quoted 4 lines
The natural thing to do is move these to
raw_object_storetoo (andrepo_submodule_initneeds to check submodule's config file). But one may argue these should be per-process instead of per-repo though. I don't know. But I thought I should mention this.For now a process and a repository is the same as git-gc or git-repack
I think you're thinking about the pack writing part, but these are configuration for pack reading (aka "object store"). If you read a blob from a submodule (e.g. git-grep), you'll use these configurations at some point.
There are of course more configuration for pack writing (like zlib_compression_level) which I deliberately avoided to mention because I don't even know where they belong.
Quoted 5 lines
doesn't know about the --recurse-submodules flag, yet.
I wonder if we ever want to teach those commands the submodule recursion, because of the issue you bring up, which settings do we apply for a submodule? Traditionally we'd just have the command line override the configuration, which I don't know if it is a good idea for these settings.
Signature · 1 line
Duy
Quoted 3 lines · On Fri, 23 Feb 2018 16:47:27 -0800 Stefan Beller <[email protected]> wrote:
v4:
* addressed feedback from the single patches (mostly nits)
* rebased on latest master
The patches I looked at previously (patches 7, 15, 19, 22, and 24) look good to me.