Skip to contentSkip to conversation

Moving global state into the repository object (part 2)

PATCH In review Stefan Beller 5 participants 23 messages Feb 28, 2018

lore
Share & tools
lore
#0 lore raw

cover Moving global state into the repository object (part 2)

This applies on top of origin/sb/object-store and is the continuation of that series, adding the repository as a context argument to functions.

This series focusses on packfile handling, exposing (re)prepare_packed_git and find_pack_entry to a repository argument.

Looking at the diffstat of "Delete ignore_env member in struct repository"[1] and "Fix initializing the_hash_algo"[2], which also build on origin/sb/object-store, this series looks rather orthogonal to those, so I would not a lot of merge conflicts.

[1] https://public-inbox.org/git/[email protected]/
[2] https://public-inbox.org/git/[email protected]/

The third series (after this one) will focus on object replacement, such that we can migrate sha1_object_info_extended at the end of the third series.

Thanks,
Stefan

Stefan Beller (11):
packfile: allow prepare_packed_git_mru to handle arbitrary
repositories
packfile: allow rearrange_packed_git to handle arbitrary repositories
packfile: allow install_packed_git to handle arbitrary repositories
packfile: add repository argument to prepare_packed_git_one
packfile: add repository argument to prepare_packed_git
packfile: add repository argument to reprepare_packed_git
packfile: allow prepare_packed_git_one to handle arbitrary
repositories
packfile: allow prepare_packed_git to handle arbitrary repositories
packfile: allow reprepare_packed_git to handle arbitrary repositories
packfile: add repository argument to find_pack_entry
packfile: allow find_pack_entry to handle arbitrary repositories

 builtin/count-objects.c  |  2 +-
 builtin/fsck.c           |  2 +-
 builtin/gc.c             |  4 +--
 builtin/pack-objects.c   |  2 +-
 builtin/pack-redundant.c |  2 +-
 builtin/receive-pack.c   |  3 +-
 bulk-checkin.c           |  3 +-
 fast-import.c            |  4 +--
 fetch-pack.c             |  3 +-
 http-backend.c           |  2 +-
 http.c                   |  2 +-
 pack-bitmap.c            |  2 +-
 packfile.c               | 72 +++++++++++++++++++---------------------
 packfile.h               | 12 ++++---
 server-info.c            |  2 +-
 sha1_file.c              |  8 ++---
 sha1_name.c              |  4 +--
 17 files changed, 66 insertions(+), 63 deletions(-)
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#1 lore raw

PATCH 1/11 packfile: allow prepare_packed_git_mru to handle arbitrary repositories

This conversion was done without the #define trick used in the earlier series refactoring to have better repository access, because this function is easy to review, as all lines are converted and it has only one caller

---
 packfile.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
packfile.c +5 -5
index ccfe084642b..b844b653265 100644
@@ -873,14 +873,14 @@ static void rearrange_packed_git(void)
 		set_next_packed_git, sort_pack);
 }
 
-static void prepare_packed_git_mru(void)
+static void prepare_packed_git_mru(struct repository *r)
 {
 	struct packed_git *p;
 
-	INIT_LIST_HEAD(&the_repository->objects.packed_git_mru);
+	INIT_LIST_HEAD(&r->objects.packed_git_mru);
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
-		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
+	for (p = r->objects.packed_git; p; p = p->next)
+		list_add_tail(&p->mru, &r->objects.packed_git_mru);
 }
 
 void prepare_packed_git(void)
@@ -894,7 +894,7 @@ void prepare_packed_git(void)
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
-	prepare_packed_git_mru();
+	prepare_packed_git_mru(the_repository);
 	the_repository->objects.packed_git_initialized = 1;
 }
 
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#2 lore raw

PATCH 2/11 packfile: allow rearrange_packed_git to handle arbitrary repositories

In reply to Stefan Beller
---
 packfile.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
packfile.c +4 -4
index b844b653265..5ccce419354 100644
@@ -866,10 +866,10 @@ static int sort_pack(const void *a_, const void *b_)
 	return -1;
 }
 
-static void rearrange_packed_git(void)
+static void rearrange_packed_git(struct repository *r)
 {
-	the_repository->objects.packed_git = llist_mergesort(
-		the_repository->objects.packed_git, get_next_packed_git,
+	r->objects.packed_git = llist_mergesort(
+		r->objects.packed_git, get_next_packed_git,
 		set_next_packed_git, sort_pack);
 }
 
@@ -893,7 +893,7 @@ void prepare_packed_git(void)
 	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
-	rearrange_packed_git();
+	rearrange_packed_git(the_repository);
 	prepare_packed_git_mru(the_repository);
 	the_repository->objects.packed_git_initialized = 1;
 }
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#3 lore raw

PATCH 3/11 packfile: allow install_packed_git to handle arbitrary repositories

In reply to Stefan Beller

This conversion was done without the #define trick used in the earlier series refactoring to have better repository access, because this function is easy to review, as it only has one caller and all lines but the first two are converted.

We must not convert 'pack_open_fds' to be a repository specific variable, as it is used to monitor resource usage of the machine that Git executes on.

---
 fast-import.c | 2 +-
 http.c        | 2 +-
 packfile.c    | 8 ++++----
 packfile.h    | 2 +-
 4 files changed, 7 insertions(+), 7 deletions(-)
fast-import.c +1 -1
index 1685fe59a20..67550584da4 100644
@@ -1038,7 +1038,7 @@ static void end_packfile(void)
 		if (!new_p)
 			die("core git rejected index %s", idx_name);
 		all_packs[pack_id] = new_p;
-		install_packed_git(new_p);
+		install_packed_git(the_repository, new_p);
 		free(idx_name);
 
 		/* Print the boundary */
http.c +1 -1
index afe2707e86b..f3c2302f84b 100644
@@ -2134,7 +2134,7 @@ int finish_http_pack_request(struct http_pack_request *preq)
 		return -1;
 	}
 
-	install_packed_git(p);
+	install_packed_git(the_repository, p);
 	free(tmp_idx);
 	return 0;
 }
packfile.c +4 -4
index 5ccce419354..1c24b02cc84 100644
@@ -680,13 +680,13 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
 	return p;
 }
 
-void install_packed_git(struct packed_git *pack)
+void install_packed_git(struct repository *r, struct packed_git *pack)
 {
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = the_repository->objects.packed_git;
-	the_repository->objects.packed_git = pack;
+	pack->next = r->objects.packed_git;
+	r->objects.packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -782,7 +782,7 @@ static void prepare_packed_git_one(char *objdir, int local)
 			     * corresponding .pack file that we can map.
 			     */
 			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
-				install_packed_git(p);
+				install_packed_git(the_repository, p);
 		}
 
 		if (!report_garbage)
packfile.h +1 -1
index 6a2c57045ca..cc3bf67ab50 100644
@@ -36,7 +36,7 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
 extern void prepare_packed_git(void);
 extern void reprepare_packed_git(void);
-extern void install_packed_git(struct packed_git *pack);
+extern void install_packed_git(struct repository *r, struct packed_git *pack);
 
 /*
  * Give a rough count of objects in the repository. This sacrifices accuracy
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#4 lore raw

Re: [PATCH 03/11] packfile: allow install_packed_git to handle arbitrary repositories

packfile.c struct packed_git *add_packed_git(const char *path, size_t path_len, int local) · line 683

Quoted 3 lines · Stefan Beller
packfile.c +1 -1
-void install_packed_git(struct packed_git *pack)
+void install_packed_git(struct repository *r, struct packed_git *pack)

This is a good thing to do. I'm just making note that this will collide with the new instances of install_packed_git() that I added in the serialized git commit graph series, specifically when adding the --stdin-packs argument [1].

Thanks,
-Stolee

[1] https://public-inbox.org/git/CAGZ79kYM0fHiYQ2+k5__A2hY1PeCyigYf3n9ZBJSKH8yJZOF0A@mail.gmail.com/T/#u

#5 lore raw

PATCH 4/11 packfile: add repository argument to prepare_packed_git_one

In reply to Stefan Beller
---
 packfile.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
packfile.c +4 -3
index 1c24b02cc84..8bb158fc84e 100644
@@ -735,7 +735,8 @@ static void report_pack_garbage(struct string_list *list)
 	report_helper(list, seen_bits, first, list->nr);
 }
 
-static void prepare_packed_git_one(char *objdir, int local)
+#define prepare_packed_git_one(r, o, l) prepare_packed_git_one_##r(o, l)
+static void prepare_packed_git_one_the_repository(char *objdir, int local)
 {
 	struct strbuf path = STRBUF_INIT;
 	size_t dirnamelen;
@@ -889,10 +890,10 @@ void prepare_packed_git(void)
 
 	if (the_repository->objects.packed_git_initialized)
 		return;
-	prepare_packed_git_one(get_object_directory(), 1);
+	prepare_packed_git_one(the_repository, get_object_directory(), 1);
 	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
-		prepare_packed_git_one(alt->path, 0);
+		prepare_packed_git_one(the_repository, alt->path, 0);
 	rearrange_packed_git(the_repository);
 	prepare_packed_git_mru(the_repository);
 	the_repository->objects.packed_git_initialized = 1;
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#6 lore raw

PATCH 5/11 packfile: add repository argument to prepare_packed_git

In reply to Stefan Beller

Add a repository argument to allow prepare_packed_git callers to be more specific about which repository to handle. See c28d027a52c (sha1_file: add repository argument to link_alt_odb_entry, 2018-02-20) for an explanation of the #define trick.

---
 builtin/count-objects.c  |  2 +-
 builtin/fsck.c           |  2 +-
 builtin/gc.c             |  2 +-
 builtin/pack-objects.c   |  2 +-
 builtin/pack-redundant.c |  2 +-
 fast-import.c            |  2 +-
 http-backend.c           |  2 +-
 pack-bitmap.c            |  2 +-
 packfile.c               | 10 +++++-----
 packfile.h               |  3 ++-
 server-info.c            |  2 +-
 sha1_name.c              |  4 ++--
 12 files changed, 18 insertions(+), 17 deletions(-)
builtin/count-objects.c +1 -1
index 9334648dccf..b262327bf6d 100644
@@ -123,7 +123,7 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
 		if (!the_repository->objects.packed_git)
-			prepare_packed_git();
+			prepare_packed_git(the_repository);
 		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local)
 				continue;
builtin/fsck.c +1 -1
index 76c94e9b5af..27f580352c5 100644
@@ -727,7 +727,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			uint32_t total = 0, count = 0;
 			struct progress *progress = NULL;
 
-			prepare_packed_git();
+			prepare_packed_git(the_repository);
 
 			if (show_progress) {
 				for (p = the_repository->objects.packed_git; p;
builtin/gc.c +1 -1
index 96ff7908677..97f34ae9fe0 100644
@@ -174,7 +174,7 @@ static int too_many_packs(void)
 	if (gc_auto_pack_limit <= 0)
 		return 0;
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
builtin/pack-objects.c +1 -1
index 7f376c2aefb..84e940a53ba 100644
@@ -3151,7 +3151,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	if (progress && all_progress_implied)
 		progress = 2;
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	if (ignore_packed_keep) {
 		struct packed_git *p;
 		for (p = the_repository->objects.packed_git; p; p = p->next)
builtin/pack-redundant.c +1 -1
index 55462bc826a..83d2395dae9 100644
@@ -631,7 +631,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)
 			break;
 	}
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 
 	if (load_all_packs)
 		load_all();
fast-import.c +1 -1
index 67550584da4..39cd0b7540d 100644
@@ -3471,7 +3471,7 @@ int cmd_main(int argc, const char **argv)
 		rc_free[i].next = &rc_free[i + 1];
 	rc_free[cmd_save - 1].next = NULL;
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	start_packfile();
 	set_die_routine(die_nicely);
 	set_checkpoint_signal();
http-backend.c +1 -1
index 4d93126c15f..4950078c936 100644
@@ -519,7 +519,7 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 	size_t cnt = 0;
 
 	select_getanyfile(hdr);
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
pack-bitmap.c +1 -1
index d51172b1d5b..273bdfb631f 100644
@@ -336,7 +336,7 @@ static int open_pack_bitmap(void)
 
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
packfile.c +5 -5
index 8bb158fc84e..6e8b98ab405 100644
@@ -817,7 +817,7 @@ unsigned long approximate_object_count(void)
 		unsigned long count;
 		struct packed_git *p;
 
-		prepare_packed_git();
+		prepare_packed_git(the_repository);
 		count = 0;
 		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (open_pack_index(p))
@@ -884,7 +884,7 @@ static void prepare_packed_git_mru(struct repository *r)
 		list_add_tail(&p->mru, &r->objects.packed_git_mru);
 }
 
-void prepare_packed_git(void)
+void prepare_packed_git_the_repository(void)
 {
 	struct alternate_object_database *alt;
 
@@ -903,7 +903,7 @@ void reprepare_packed_git(void)
 {
 	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 }
 
 unsigned long unpack_object_header_buffer(const unsigned char *buf,
@@ -1844,7 +1844,7 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 {
 	struct list_head *pos;
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	if (!the_repository->objects.packed_git)
 		return 0;
 
@@ -1897,7 +1897,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int r = 0;
 	int pack_errors = 0;
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
packfile.h +2 -1
index cc3bf67ab50..39eb590b2ae 100644
@@ -34,7 +34,8 @@ extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_
 #define PACKDIR_FILE_GARBAGE 4
 extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
-extern void prepare_packed_git(void);
+#define prepare_packed_git(r) prepare_packed_git_##r()
+extern void prepare_packed_git_the_repository(void);
 extern void reprepare_packed_git(void);
 extern void install_packed_git(struct repository *r, struct packed_git *pack);
 
server-info.c +1 -1
index 96885c45f64..6cd1506e5e3 100644
@@ -201,7 +201,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdir = get_object_directory();
 	objdirlen = strlen(objdir);
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
sha1_name.c +2 -2
index 3e490ee8f62..d3769713ea0 100644
@@ -196,7 +196,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 {
 	struct packed_git *p;
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
 	     p = p->next)
 		unique_in_pack(p, ds);
@@ -567,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 {
 	struct packed_git *p;
 
-	prepare_packed_git();
+	prepare_packed_git(the_repository);
 	for (p = the_repository->objects.packed_git; p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#7 lore raw

PATCH 6/11 packfile: add repository argument to reprepare_packed_git

In reply to Stefan Beller

See previous patch for explanation.

---
 builtin/gc.c           | 2 +-
 builtin/receive-pack.c | 3 ++-
 bulk-checkin.c         | 3 ++-
 fetch-pack.c           | 3 ++-
 packfile.c             | 2 +-
 packfile.h             | 3 ++-
 sha1_file.c            | 2 +-
 7 files changed, 11 insertions(+), 7 deletions(-)
builtin/gc.c +1 -1
index 97f34ae9fe0..c16020ef42a 100644
@@ -478,7 +478,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
 		return error(FAILED_RUN, rerere.argv[0]);
 
 	report_garbage = report_pack_garbage;
-	reprepare_packed_git();
+	reprepare_packed_git(the_repository);
 	if (pack_garbage.nr > 0)
 		clean_pack_garbage();
 
builtin/receive-pack.c +2 -1
index 954fc72c7cb..8b03a6e03dc 100644
@@ -1,4 +1,5 @@
 #include "builtin.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
 #include "pack.h"
@@ -1778,7 +1779,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		status = finish_command(&child);
 		if (status)
 			return "index-pack abnormal exit";
-		reprepare_packed_git();
+		reprepare_packed_git(the_repository);
 	}
 	return NULL;
 }
bulk-checkin.c +2 -1
index 3310fd210a1..eadc2d51720 100644
@@ -3,6 +3,7 @@
  */
 #include "cache.h"
 #include "bulk-checkin.h"
+#include "repository.h"
 #include "csum-file.h"
 #include "pack.h"
 #include "strbuf.h"
@@ -57,7 +58,7 @@ static void finish_bulk_checkin(struct bulk_checkin_state *state)
 
 	strbuf_release(&packname);
 	/* Make objects we just wrote available to ourselves */
-	reprepare_packed_git();
+	reprepare_packed_git(the_repository);
 }
 
 static int already_written(struct bulk_checkin_state *state, unsigned char sha1[])
fetch-pack.c +2 -1
index 8253d746e0c..eac5928a27b 100644
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
 #include "refs.h"
@@ -1192,7 +1193,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,
 	prepare_shallow_info(&si, shallow);
 	ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,
 				&si, pack_lockfile);
-	reprepare_packed_git();
+	reprepare_packed_git(the_repository);
 	update_shallow(args, sought, nr_sought, &si);
 	clear_shallow_info(&si);
 	return ref_cpy;
packfile.c +1 -1
index 6e8b98ab405..6ea50230aab 100644
@@ -899,7 +899,7 @@ void prepare_packed_git_the_repository(void)
 	the_repository->objects.packed_git_initialized = 1;
 }
 
-void reprepare_packed_git(void)
+void reprepare_packed_git_the_repository(void)
 {
 	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
packfile.h +2 -1
index 39eb590b2ae..9afbf73657e 100644
@@ -36,7 +36,8 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
 #define prepare_packed_git(r) prepare_packed_git_##r()
 extern void prepare_packed_git_the_repository(void);
-extern void reprepare_packed_git(void);
+#define reprepare_packed_git(r) reprepare_packed_git_##r()
+extern void reprepare_packed_git_the_repository(void);
 extern void install_packed_git(struct repository *r, struct packed_git *pack);
 
 /*
sha1_file.c +1 -1
index 36282acb1a8..0b9fefaaf02 100644
@@ -1281,7 +1281,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 			return 0;
 
 		/* Not a loose object; someone else may have just packed it. */
-		reprepare_packed_git();
+		reprepare_packed_git(the_repository);
 		if (find_pack_entry(real, &e))
 			break;
 
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#8 lore raw

PATCH 7/11 packfile: allow prepare_packed_git_one to handle arbitrary repositories

In reply to Stefan Beller
---
 packfile.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)
packfile.c +3 -4
index 6ea50230aab..c45516acd41 100644
@@ -735,8 +735,7 @@ static void report_pack_garbage(struct string_list *list)
 	report_helper(list, seen_bits, first, list->nr);
 }
 
-#define prepare_packed_git_one(r, o, l) prepare_packed_git_one_##r(o, l)
-static void prepare_packed_git_one_the_repository(char *objdir, int local)
+static void prepare_packed_git_one(struct repository *r, char *objdir, int local)
 {
 	struct strbuf path = STRBUF_INIT;
 	size_t dirnamelen;
@@ -769,7 +768,7 @@ static void prepare_packed_git_one_the_repository(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = the_repository->objects.packed_git; p;
+			for (p = r->objects.packed_git; p;
 			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
@@ -783,7 +782,7 @@ static void prepare_packed_git_one_the_repository(char *objdir, int local)
 			     * corresponding .pack file that we can map.
 			     */
 			    (p = add_packed_git(path.buf, path.len, local)) != NULL)
-				install_packed_git(the_repository, p);
+				install_packed_git(r, p);
 		}
 
 		if (!report_garbage)
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#9 lore raw

PATCH 8/11 packfile: allow prepare_packed_git to handle arbitrary repositories

In reply to Stefan Beller
---
 packfile.c | 18 +++++++++---------
 packfile.h |  3 +--
 2 files changed, 10 insertions(+), 11 deletions(-)
packfile.c +9 -9
index c45516acd41..9a3efc01555 100644
@@ -883,19 +883,19 @@ static void prepare_packed_git_mru(struct repository *r)
 		list_add_tail(&p->mru, &r->objects.packed_git_mru);
 }
 
-void prepare_packed_git_the_repository(void)
+void prepare_packed_git(struct repository *r)
 {
 	struct alternate_object_database *alt;
 
-	if (the_repository->objects.packed_git_initialized)
+	if (r->objects.packed_git_initialized)
 		return;
-	prepare_packed_git_one(the_repository, get_object_directory(), 1);
-	prepare_alt_odb(the_repository);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
-		prepare_packed_git_one(the_repository, alt->path, 0);
-	rearrange_packed_git(the_repository);
-	prepare_packed_git_mru(the_repository);
-	the_repository->objects.packed_git_initialized = 1;
+	prepare_packed_git_one(r, get_object_directory(), 1);
+	prepare_alt_odb(r);
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next)
+		prepare_packed_git_one(r, alt->path, 0);
+	rearrange_packed_git(r);
+	prepare_packed_git_mru(r);
+	r->objects.packed_git_initialized = 1;
 }
 
 void reprepare_packed_git_the_repository(void)
packfile.h +1 -2
index 9afbf73657e..9142866c8ae 100644
@@ -34,8 +34,7 @@ extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_
 #define PACKDIR_FILE_GARBAGE 4
 extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
-#define prepare_packed_git(r) prepare_packed_git_##r()
-extern void prepare_packed_git_the_repository(void);
+extern void prepare_packed_git(struct repository *r);
 #define reprepare_packed_git(r) reprepare_packed_git_##r()
 extern void reprepare_packed_git_the_repository(void);
 extern void install_packed_git(struct repository *r, struct packed_git *pack);
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#10 lore raw

PATCH 9/11 packfile: allow reprepare_packed_git to handle arbitrary repositories

In reply to Stefan Beller
---
 packfile.c | 8 ++++----
 packfile.h | 3 +--
 2 files changed, 5 insertions(+), 6 deletions(-)
packfile.c +4 -4
index 9a3efc01555..b12fc0789e0 100644
@@ -898,11 +898,11 @@ void prepare_packed_git(struct repository *r)
 	r->objects.packed_git_initialized = 1;
 }
 
-void reprepare_packed_git_the_repository(void)
+void reprepare_packed_git(struct repository *r)
 {
-	the_repository->objects.approximate_object_count_valid = 0;
-	the_repository->objects.packed_git_initialized = 0;
-	prepare_packed_git(the_repository);
+	r->objects.approximate_object_count_valid = 0;
+	r->objects.packed_git_initialized = 0;
+	prepare_packed_git(r);
 }
 
 unsigned long unpack_object_header_buffer(const unsigned char *buf,
packfile.h +1 -2
index 9142866c8ae..99f4741bd95 100644
@@ -35,8 +35,7 @@ extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_
 extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
 extern void prepare_packed_git(struct repository *r);
-#define reprepare_packed_git(r) reprepare_packed_git_##r()
-extern void reprepare_packed_git_the_repository(void);
+extern void reprepare_packed_git(struct repository *r);
 extern void install_packed_git(struct repository *r, struct packed_git *pack);
 
 /*
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#11 lore raw

PATCH 10/11 packfile: add repository argument to find_pack_entry

In reply to Stefan Beller

While at it move the documentation to the header and mention which pack files are searched.

---
 packfile.c  | 8 ++------
 packfile.h  | 7 ++++++-
 sha1_file.c | 6 +++---
 3 files changed, 11 insertions(+), 10 deletions(-)
packfile.c +2 -6
index b12fc0789e0..e16f8252233 100644
@@ -1835,11 +1835,7 @@ static int fill_pack_entry(const unsigned char *sha1,
 	return 1;
 }
 
-/*
- * Iff a pack file contains the object named by sha1, return true and
- * store its location to e.
- */
-int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
+int find_pack_entry_the_repository(const unsigned char *sha1, struct pack_entry *e)
 {
 	struct list_head *pos;
 
@@ -1860,7 +1856,7 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 int has_sha1_pack(const unsigned char *sha1)
 {
 	struct pack_entry e;
-	return find_pack_entry(sha1, &e);
+	return find_pack_entry(the_repository, sha1, &e);
 }
 
 int has_pack_index(const unsigned char *sha1)
packfile.h +6 -1
index 99f4741bd95..feeabd6f031 100644
@@ -120,7 +120,12 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct obje
 extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);
 extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);
 
-extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);
+/*
+ * Iff a pack file in the given repository contains the object named by sha1,
+ * return true and store its location to e.
+ */
+#define find_pack_entry(r, s, e) find_pack_entry_##r(s, e)
+extern int find_pack_entry_the_repository(const unsigned char *sha1, struct pack_entry *e);
 
 extern int has_sha1_pack(const unsigned char *sha1);
 
sha1_file.c +3 -3
index 0b9fefaaf02..d498b3e90b9 100644
@@ -1273,7 +1273,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 	}
 
 	while (1) {
-		if (find_pack_entry(real, &e))
+		if (find_pack_entry(the_repository, real, &e))
 			break;
 
 		/* Most likely it's a loose object. */
@@ -1282,7 +1282,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 
 		/* Not a loose object; someone else may have just packed it. */
 		reprepare_packed_git(the_repository);
-		if (find_pack_entry(real, &e))
+		if (find_pack_entry(the_repository, real, &e))
 			break;
 
 		/* Check if it is a missing object */
@@ -1662,7 +1662,7 @@ static int freshen_loose_object(const unsigned char *sha1)
 static int freshen_packed_object(const unsigned char *sha1)
 {
 	struct pack_entry e;
-	if (!find_pack_entry(sha1, &e))
+	if (!find_pack_entry(the_repository, sha1, &e))
 		return 0;
 	if (e.p->freshened)
 		return 1;
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#12 lore raw

PATCH 11/11 packfile: allow find_pack_entry to handle arbitrary repositories

In reply to Stefan Beller
---
 packfile.c | 10 +++++-----
 packfile.h |  3 +--
 2 files changed, 6 insertions(+), 7 deletions(-)
packfile.c +5 -5
index e16f8252233..399c59e640f 100644
@@ -1835,18 +1835,18 @@ static int fill_pack_entry(const unsigned char *sha1,
 	return 1;
 }
 
-int find_pack_entry_the_repository(const unsigned char *sha1, struct pack_entry *e)
+int find_pack_entry(struct repository *r, const unsigned char *sha1, struct pack_entry *e)
 {
 	struct list_head *pos;
 
-	prepare_packed_git(the_repository);
-	if (!the_repository->objects.packed_git)
+	prepare_packed_git(r);
+	if (!r->objects.packed_git)
 		return 0;
 
-	list_for_each(pos, &the_repository->objects.packed_git_mru) {
+	list_for_each(pos, &r->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		if (fill_pack_entry(sha1, e, p)) {
-			list_move(&p->mru, &the_repository->objects.packed_git_mru);
+			list_move(&p->mru, &r->objects.packed_git_mru);
 			return 1;
 		}
 	}
packfile.h +1 -2
index feeabd6f031..6f7b9e91d66 100644
@@ -124,8 +124,7 @@ extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);
  * Iff a pack file in the given repository contains the object named by sha1,
  * return true and store its location to e.
  */
-#define find_pack_entry(r, s, e) find_pack_entry_##r(s, e)
-extern int find_pack_entry_the_repository(const unsigned char *sha1, struct pack_entry *e);
+extern int find_pack_entry(struct repository *r, const unsigned char *sha1, struct pack_entry *e);
 
 extern int has_sha1_pack(const unsigned char *sha1);
 
Signature · 1 line
2.16.2.395.g2e18187dfd-goog
#13 lore raw
In reply to Stefan Beller
Quoted 5 lines · Stefan Beller

This applies on top of origin/sb/object-store and is the continuation of that series, adding the repository as a context argument to functions.

This series focusses on packfile handling, exposing (re)prepare_packed_git and find_pack_entry to a repository argument.

It looks good.

Looking at the full-series diff though, it makes me wonder if we should keep prepare_packed_git() private (i.e. how we initialize the object store, packfile included, is a private matter). How about something like this on top?

-- 8< --
Subject: [PATCH] packfile: keep raw_object_store.packed_git{,_mru} fields private

These fields are initialized lazily via prepare_packed_git(). All access to these must call that function first but unless you know the implementation details, you may not see the connection.

Keep that connection internal. These fields should only be accessed via get_packed_git() and get_packed_git_mru() outside packfile.c.

---
 builtin/count-objects.c  |  4 +---
 builtin/fsck.c           |  6 ++----
 builtin/gc.c             |  3 +--
 builtin/pack-objects.c   | 21 +++++++++++----------
 builtin/pack-redundant.c |  6 ++----
 fast-import.c            |  7 ++-----
 http-backend.c           |  5 ++---
 object-store.h           |  4 ++--
 pack-bitmap.c            |  3 +--
 packfile.c               | 15 ++++++++++++++-
 packfile.h               |  6 +++++-
 server-info.c            |  5 ++---
 sha1_name.c              |  6 ++----
 13 files changed, 47 insertions(+), 44 deletions(-)
builtin/count-objects.c +1 -3
index d480301763..088309945b 100644
@@ -121,9 +121,7 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!the_repository->objects.packed_git)
-			prepare_packed_git(the_repository);
-		for (p = the_repository->objects.packed_git; p; p = p->next) {
+		for (p = get_packed_git(the_repository); p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
builtin/fsck.c +2 -4
index 0a043a43c2..6d86f2581a 100644
@@ -732,10 +732,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			uint32_t total = 0, count = 0;
 			struct progress *progress = NULL;
 
-			prepare_packed_git(the_repository);
-
 			if (show_progress) {
-				for (p = the_repository->objects.packed_git; p;
+				for (p = get_packed_git(the_repository); p;
 				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
@@ -744,7 +742,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = the_repository->objects.packed_git; p;
+			for (p = get_packed_git(the_repository); p;
 			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
builtin/gc.c +1 -2
index 80d19c54d5..be63bec09c 100644
@@ -173,8 +173,7 @@ static int too_many_packs(void)
 	if (gc_auto_pack_limit <= 0)
 		return 0;
 
-	prepare_packed_git(the_repository);
-	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
+	for (cnt = 0, p = get_packed_git(the_repository); p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
builtin/pack-objects.c +11 -10
index 5e2590f882..a305f50100 100644
@@ -1026,7 +1026,7 @@ static int want_object_in_pack(const struct object_id *oid,
 		if (want != -1)
 			return want;
 	}
-	list_for_each(pos, &the_repository->objects.packed_git_mru) {
+	list_for_each(pos, get_packed_git_mru(the_repository)) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		off_t offset;
 
@@ -1045,7 +1045,7 @@ static int want_object_in_pack(const struct object_id *oid,
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
 				list_move(&p->mru,
-					  &the_repository->objects.packed_git_mru);
+					  get_packed_git_mru(the_repository));
 			if (want != -1)
 				return want;
 		}
@@ -2674,7 +2674,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2737,8 +2737,10 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	static struct packed_git *last_found = (void *)1;
 	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found :
-					the_repository->objects.packed_git;
+	if (last_found != (void *)1)
+		p = last_found;
+	else
+		p = get_packed_git(the_repository);
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2747,7 +2749,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = the_repository->objects.packed_git;
+			p = get_packed_git(the_repository);
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2783,7 +2785,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3151,10 +3153,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	if (progress && all_progress_implied)
 		progress = 2;
 
-	prepare_packed_git(the_repository);
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = the_repository->objects.packed_git; p; p = p->next)
+		for (p = get_packed_git(the_repository); p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3167,7 +3168,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = the_repository->objects.packed_git; p; p = p->next) {
+		for (p = get_packed_git(the_repository); p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
builtin/pack-redundant.c +2 -4
index 3b02f94248..02b5f0becc 100644
@@ -572,7 +572,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = the_repository->objects.packed_git;
+	struct packed_git *p = get_packed_git(the_repository);
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -587,7 +587,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = the_repository->objects.packed_git;
+	struct packed_git *p = get_packed_git(the_repository);
 
 	while (p) {
 		add_pack(p);
@@ -630,8 +630,6 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)
 			break;
 	}
 
-	prepare_packed_git(the_repository);
-
 	if (load_all_packs)
 		load_all();
 	else
fast-import.c +2 -5
index f2e255ed7e..041217eedf 100644
@@ -1111,8 +1111,7 @@ static int store_object(
 	if (e->idx.offset) {
 		duplicate_count_by_type[type]++;
 		return 1;
-	} else if (find_sha1_pack(oid.hash,
-				  the_repository->objects.packed_git)) {
+	} else if (find_sha1_pack(oid.hash, get_packed_git(the_repository))) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1309,8 +1308,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		duplicate_count_by_type[OBJ_BLOB]++;
 		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash,
-				  the_repository->objects.packed_git)) {
+	} else if (find_sha1_pack(oid.hash, get_packed_git(the_repository))) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -3474,7 +3472,6 @@ int cmd_main(int argc, const char **argv)
 		rc_free[i].next = &rc_free[i + 1];
 	rc_free[cmd_save - 1].next = NULL;
 
-	prepare_packed_git(the_repository);
 	start_packfile();
 	set_die_routine(die_nicely);
 	set_checkpoint_signal();
http-backend.c +2 -3
index defa6ba350..22d2e1668e 100644
@@ -518,14 +518,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 	size_t cnt = 0;
 
 	select_getanyfile(hdr);
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
object-store.h +2 -2
index afe2f93459..636375753f 100644
@@ -87,9 +87,9 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
-	struct packed_git *packed_git;
+	struct packed_git *packed_git; /* private */
 	/* A most-recently-used ordered version of the packed_git list. */
-	struct list_head packed_git_mru;
+	struct list_head packed_git_mru; /* private */
 
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
pack-bitmap.c +1 -2
index bcc04bc45e..2a007b5539 100644
@@ -335,8 +335,7 @@ static int open_pack_bitmap(void)
 
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
packfile.c +14 -1
index d9065ccd32..ed26ab1b3b 100644
@@ -802,6 +802,7 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local
 	strbuf_release(&path);
 }
 
+static void prepare_packed_git(struct repository *r);
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -882,7 +883,7 @@ static void prepare_packed_git_mru(struct repository *r)
 		list_add_tail(&p->mru, &r->objects.packed_git_mru);
 }
 
-void prepare_packed_git(struct repository *r)
+static void prepare_packed_git(struct repository *r)
 {
 	struct alternate_object_database *alt;
 
@@ -897,6 +898,18 @@ void prepare_packed_git(struct repository *r)
 	r->objects.packed_git_initialized = 1;
 }
 
+struct packed_git *get_packed_git(struct repository *r)
+{
+	prepare_packed_git(r);
+	return r->objects.packed_git;
+}
+
+struct list_head *get_packed_git_mru(struct repository *r)
+{
+	prepare_packed_git(r);
+	return &r->objects.packed_git_mru;
+}
+
 void reprepare_packed_git(struct repository *r)
 {
 	r->objects.approximate_object_count_valid = 0;
packfile.h +5 -1
index 6f7b9e91d6..1903984e22 100644
@@ -3,6 +3,9 @@
 
 #include "oidset.h"
 
+struct list_head;
+struct packed_git;
+
 /*
  * Generate the filename to be used for a pack file with checksum "sha1" and
  * extension "ext". The result is written into the strbuf "buf", overwriting
@@ -34,7 +37,8 @@ extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_
 #define PACKDIR_FILE_GARBAGE 4
 extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
-extern void prepare_packed_git(struct repository *r);
+extern struct packed_git *get_packed_git(struct repository *r);
+extern struct list_head *get_packed_git_mru(struct repository *r);
 extern void reprepare_packed_git(struct repository *r);
 extern void install_packed_git(struct repository *r, struct packed_git *pack);
 
server-info.c +2 -3
index f5e4d1cc89..af737acd24 100644
@@ -200,8 +200,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdir = get_object_directory();
 	objdirlen = strlen(objdir);
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -211,7 +210,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
+	for (i = 0, p = get_packed_git(the_repository); p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
sha1_name.c +2 -4
index 773322ac81..f25a0970ca 100644
@@ -195,8 +195,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 {
 	struct packed_git *p;
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	for (p = get_packed_git(the_repository); p && !ds->ambiguous;
 	     p = p->next)
 		unique_in_pack(p, ds);
 }
@@ -566,8 +565,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 {
 	struct packed_git *p;
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = get_packed_git(the_repository); p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
Signature · 4 lines
2.16.1.399.g632f88eed1


-- 8< --
#14 lore raw
Quoted 7 lines · Duy Nguyen
Quoted 5 lines · Stefan Beller

This applies on top of origin/sb/object-store and is the continuation of that series, adding the repository as a context argument to functions.

This series focusses on packfile handling, exposing (re)prepare_packed_git and find_pack_entry to a repository argument.

It looks good.

I agree.

Quoted 4 lines

Looking at the full-series diff though, it makes me wonder if we should keep prepare_packed_git() private (i.e. how we initialize the object store, packfile included, is a private matter). How about something like this on top?

I think the get_packed_git() approach is cleaner than navigating directly to the_repository->objects.packed_git that you expect to be initialized by an earlier method call.

Thanks,
-Stolee

Quoted 389 lines

-- 8< --
Subject: [PATCH] packfile: keep raw_object_store.packed_git{,_mru} fields private

These fields are initialized lazily via prepare_packed_git(). All access to these must call that function first but unless you know the implementation details, you may not see the connection.

Keep that connection internal. These fields should only be accessed via get_packed_git() and get_packed_git_mru() outside packfile.c.

---
  builtin/count-objects.c  |  4 +---
  builtin/fsck.c           |  6 ++----
  builtin/gc.c             |  3 +--
  builtin/pack-objects.c   | 21 +++++++++++----------
  builtin/pack-redundant.c |  6 ++----
  fast-import.c            |  7 ++-----
  http-backend.c           |  5 ++---
  object-store.h           |  4 ++--
  pack-bitmap.c            |  3 +--
  packfile.c               | 15 ++++++++++++++-
  packfile.h               |  6 +++++-
  server-info.c            |  5 ++---
  sha1_name.c              |  6 ++----
  13 files changed, 47 insertions(+), 44 deletions(-)
builtin/count-objects.c +1 -3
index d480301763..088309945b 100644
@@ -121,9 +121,7 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
  		struct strbuf loose_buf = STRBUF_INIT;
  		struct strbuf pack_buf = STRBUF_INIT;
  		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!the_repository->objects.packed_git)
-			prepare_packed_git(the_repository);
-		for (p = the_repository->objects.packed_git; p; p = p->next) {
+		for (p = get_packed_git(the_repository); p; p = p->next) {
  			if (!p->pack_local)
  				continue;
  			if (open_pack_index(p))
builtin/fsck.c +2 -4
index 0a043a43c2..6d86f2581a 100644
@@ -732,10 +732,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
  			uint32_t total = 0, count = 0;
  			struct progress *progress = NULL;
 
-			prepare_packed_git(the_repository);
-
  			if (show_progress) {
-				for (p = the_repository->objects.packed_git; p;
+				for (p = get_packed_git(the_repository); p;
  				     p = p->next) {
  					if (open_pack_index(p))
  						continue;
@@ -744,7 +742,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
  				progress = start_progress(_("Checking objects"), total);
  			}
-			for (p = the_repository->objects.packed_git; p;
+			for (p = get_packed_git(the_repository); p;
  			     p = p->next) {
  				/* verify gives error messages itself */
  				if (verify_pack(p, fsck_obj_buffer,
builtin/gc.c +1 -2
index 80d19c54d5..be63bec09c 100644
@@ -173,8 +173,7 @@ static int too_many_packs(void)
  	if (gc_auto_pack_limit <= 0)
  		return 0;
 
-	prepare_packed_git(the_repository);
-	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
+	for (cnt = 0, p = get_packed_git(the_repository); p; p = p->next) {
  		if (!p->pack_local)
  			continue;
  		if (p->pack_keep)
builtin/pack-objects.c +11 -10
index 5e2590f882..a305f50100 100644
@@ -1026,7 +1026,7 @@ static int want_object_in_pack(const struct object_id *oid,
  		if (want != -1)
  			return want;
  	}
-	list_for_each(pos, &the_repository->objects.packed_git_mru) {
+	list_for_each(pos, get_packed_git_mru(the_repository)) {
  		struct packed_git *p = list_entry(pos, struct packed_git, mru);
  		off_t offset;
 
@@ -1045,7 +1045,7 @@ static int want_object_in_pack(const struct object_id *oid,
  			want = want_found_object(exclude, p);
  			if (!exclude && want > 0)
  				list_move(&p->mru,
-					  &the_repository->objects.packed_git_mru);
+					  get_packed_git_mru(the_repository));
  			if (want != -1)
  				return want;
  		}
@@ -2674,7 +2674,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
  	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
  		struct object_id oid;
  		struct object *o;
 
@@ -2737,8 +2737,10 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
  	static struct packed_git *last_found = (void *)1;
  	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found :
-					the_repository->objects.packed_git;
+	if (last_found != (void *)1)
+		p = last_found;
+	else
+		p = get_packed_git(the_repository);
 
  	while (p) {
  		if ((!p->pack_local || p->pack_keep) &&
@@ -2747,7 +2749,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
  			return 1;
  		}
  		if (p == last_found)
-			p = the_repository->objects.packed_git;
+			p = get_packed_git(the_repository);
  		else
  			p = p->next;
  		if (p == last_found)
@@ -2783,7 +2785,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
  	uint32_t i;
  	struct object_id oid;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
  		if (!p->pack_local || p->pack_keep)
  			continue;
 
@@ -3151,10 +3153,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
  	if (progress && all_progress_implied)
  		progress = 2;
 
-	prepare_packed_git(the_repository);
  	if (ignore_packed_keep) {
  		struct packed_git *p;
-		for (p = the_repository->objects.packed_git; p; p = p->next)
+		for (p = get_packed_git(the_repository); p; p = p->next)
  			if (p->pack_local && p->pack_keep)
  				break;
  		if (!p) /* no keep-able packs found */
@@ -3167,7 +3168,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
  		 * also covers non-local objects
  		 */
  		struct packed_git *p;
-		for (p = the_repository->objects.packed_git; p; p = p->next) {
+		for (p = get_packed_git(the_repository); p; p = p->next) {
  			if (!p->pack_local) {
  				have_non_local_packs = 1;
  				break;
builtin/pack-redundant.c +2 -4
index 3b02f94248..02b5f0becc 100644
@@ -572,7 +572,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
  static struct pack_list * add_pack_file(const char *filename)
  {
-	struct packed_git *p = the_repository->objects.packed_git;
+	struct packed_git *p = get_packed_git(the_repository);
 
  	if (strlen(filename) < 40)
  		die("Bad pack filename: %s", filename);
@@ -587,7 +587,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
  static void load_all(void)
  {
-	struct packed_git *p = the_repository->objects.packed_git;
+	struct packed_git *p = get_packed_git(the_repository);
 
  	while (p) {
  		add_pack(p);
@@ -630,8 +630,6 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix)
  			break;
  	}
 
-	prepare_packed_git(the_repository);
-
  	if (load_all_packs)
  		load_all();
  	else
fast-import.c +2 -5
index f2e255ed7e..041217eedf 100644
@@ -1111,8 +1111,7 @@ static int store_object(
  	if (e->idx.offset) {
  		duplicate_count_by_type[type]++;
  		return 1;
-	} else if (find_sha1_pack(oid.hash,
-				  the_repository->objects.packed_git)) {
+	} else if (find_sha1_pack(oid.hash, get_packed_git(the_repository))) {
  		e->type = type;
  		e->pack_id = MAX_PACK_ID;
  		e->idx.offset = 1; /* just not zero! */
@@ -1309,8 +1308,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
  		duplicate_count_by_type[OBJ_BLOB]++;
  		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash,
-				  the_repository->objects.packed_git)) {
+	} else if (find_sha1_pack(oid.hash, get_packed_git(the_repository))) {
  		e->type = OBJ_BLOB;
  		e->pack_id = MAX_PACK_ID;
  		e->idx.offset = 1; /* just not zero! */
@@ -3474,7 +3472,6 @@ int cmd_main(int argc, const char **argv)
  		rc_free[i].next = &rc_free[i + 1];
  	rc_free[cmd_save - 1].next = NULL;
 
-	prepare_packed_git(the_repository);
  	start_packfile();
  	set_die_routine(die_nicely);
  	set_checkpoint_signal();
http-backend.c +2 -3
index defa6ba350..22d2e1668e 100644
@@ -518,14 +518,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
  	size_t cnt = 0;
 
  	select_getanyfile(hdr);
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
  		if (p->pack_local)
  			cnt++;
  	}
 
  	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
  		if (p->pack_local)
  			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
  	}
object-store.h +2 -2
index afe2f93459..636375753f 100644
@@ -87,9 +87,9 @@ struct raw_object_store {
  	 */
  	char *objectdir;
 
-	struct packed_git *packed_git;
+	struct packed_git *packed_git; /* private */
  	/* A most-recently-used ordered version of the packed_git list. */
-	struct list_head packed_git_mru;
+	struct list_head packed_git_mru; /* private */
 
  	struct alternate_object_database *alt_odb_list;
  	struct alternate_object_database **alt_odb_tail;
pack-bitmap.c +1 -2
index bcc04bc45e..2a007b5539 100644
@@ -335,8 +335,7 @@ static int open_pack_bitmap(void)
 
  	assert(!bitmap_git.map && !bitmap_git.loaded);
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
  		if (open_pack_bitmap_1(p) == 0)
  			ret = 0;
  	}
packfile.c +14 -1
index d9065ccd32..ed26ab1b3b 100644
@@ -802,6 +802,7 @@ static void prepare_packed_git_one(struct repository *r, char *objdir, int local
  	strbuf_release(&path);
  }
 
+static void prepare_packed_git(struct repository *r);
  /*
   * Give a fast, rough count of the number of objects in the repository. This
   * ignores loose objects completely. If you have a lot of them, then either
@@ -882,7 +883,7 @@ static void prepare_packed_git_mru(struct repository *r)
  		list_add_tail(&p->mru, &r->objects.packed_git_mru);
  }
 
-void prepare_packed_git(struct repository *r)
+static void prepare_packed_git(struct repository *r)
  {
  	struct alternate_object_database *alt;
 
@@ -897,6 +898,18 @@ void prepare_packed_git(struct repository *r)
  	r->objects.packed_git_initialized = 1;
  }
 
+struct packed_git *get_packed_git(struct repository *r)
+{
+	prepare_packed_git(r);
+	return r->objects.packed_git;
+}
+
+struct list_head *get_packed_git_mru(struct repository *r)
+{
+	prepare_packed_git(r);
+	return &r->objects.packed_git_mru;
+}
+
  void reprepare_packed_git(struct repository *r)
  {
  	r->objects.approximate_object_count_valid = 0;
packfile.h +5 -1
index 6f7b9e91d6..1903984e22 100644
@@ -3,6 +3,9 @@
 
  #include "oidset.h"
 
+struct list_head;
+struct packed_git;
+
  /*
   * Generate the filename to be used for a pack file with checksum "sha1" and
   * extension "ext". The result is written into the strbuf "buf", overwriting
@@ -34,7 +37,8 @@ extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_
  #define PACKDIR_FILE_GARBAGE 4
  extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
-extern void prepare_packed_git(struct repository *r);
+extern struct packed_git *get_packed_git(struct repository *r);
+extern struct list_head *get_packed_git_mru(struct repository *r);
  extern void reprepare_packed_git(struct repository *r);
  extern void install_packed_git(struct repository *r, struct packed_git *pack);
 
server-info.c +2 -3
index f5e4d1cc89..af737acd24 100644
@@ -200,8 +200,7 @@ static void init_pack_info(const char *infofile, int force)
  	objdir = get_object_directory();
  	objdirlen = strlen(objdir);
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next) {
+	for (p = get_packed_git(the_repository); p; p = p->next) {
  		/* we ignore things on alternate path since they are
  		 * not available to the pullers in general.
  		 */
@@ -211,7 +210,7 @@ static void init_pack_info(const char *infofile, int force)
  	}
  	num_pack = i;
  	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
+	for (i = 0, p = get_packed_git(the_repository); p; p = p->next) {
  		if (!p->pack_local)
  			continue;
  		info[i] = xcalloc(1, sizeof(struct pack_info));
sha1_name.c +2 -4
index 773322ac81..f25a0970ca 100644
@@ -195,8 +195,7 @@ static void find_short_packed_object(struct disambiguate_state *ds)
  {
  	struct packed_git *p;
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	for (p = get_packed_git(the_repository); p && !ds->ambiguous;
  	     p = p->next)
  		unique_in_pack(p, ds);
  }
@@ -566,8 +565,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
  {
  	struct packed_git *p;
 
-	prepare_packed_git(the_repository);
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = get_packed_git(the_repository); p; p = p->next)
  		find_abbrev_len_for_pack(p, mad);
  }
 
#15 lore raw
In reply to Duy Nguyen
Quoted 4 lines · Duy Nguyen

Looking at the full-series diff though, it makes me wonder if we should keep prepare_packed_git() private (i.e. how we initialize the object store, packfile included, is a private matter). How about something like this on top?

Yup, that looks cleaner.

#16 lore raw
Quoted 8 lines · Junio C Hamano
Quoted 4 lines · Duy Nguyen

Looking at the full-series diff though, it makes me wonder if we should keep prepare_packed_git() private (i.e. how we initialize the object store, packfile included, is a private matter). How about something like this on top?

Yup, that looks cleaner.

I agree that it looks cleaner. So we plan on just putting it on top of that series?

Thanks for the help on refactoring,
Stefan

#17 lore raw
Quoted 8 lines · Stefan Beller
Quoted 7 lines · Junio C Hamano
Quoted 4 lines · Duy Nguyen

Looking at the full-series diff though, it makes me wonder if we should keep prepare_packed_git() private (i.e. how we initialize the object store, packfile included, is a private matter). How about something like this on top?

Quoted 4 lines
Quoted 1 line

Yup, that looks cleaner.

I agree that it looks cleaner. So we plan on just putting it on top of that series?

We tend to avoid "oops, that was wrong and here is a band aid on top" for things that are still mushy, so it would be preferrable to get it fixed inline, especially if there are more changes to the other parts of the series coming.

#18 lore raw
In reply to Stefan Beller
Quoted 2 lines · Stefan Beller

This applies on top of origin/sb/object-store and is the continuation of that series, adding the repository as a context argument to functions.

Wait a minute. Is that topic ever shown to work well together with other topics in flight and are now ready to be built upon? I had an impression that it is just starting to get serious reviews.

Sorry, but I am behind ;-)

#19 lore raw
Quoted 8 lines · Junio C Hamano
Quoted 2 lines · Stefan Beller

This applies on top of origin/sb/object-store and is the continuation of that series, adding the repository as a context argument to functions.

Wait a minute. Is that topic ever shown to work well together with other topics in flight and are now ready to be built upon? I had an impression that it is just starting to get serious reviews.

And I had the impression the serious reviews were done and fine; the only issue would be demonstrating its working fine with other series, where I was also worrying about conflicts with brians series. And to address that, I'd just send series in small sizes.

Quoted 1 line

Sorry, but I am behind ;-)

Is there anything that a contributor can help with that eases refactoring series in flight?

Personally I find these refactorings not as rewarding as new features or fixing bugs, and additionally the risk of conflict rises, whereby the maintainer is likely to push back to the contributor to "please rebase", and rebasing with just textual conflicts is also not rewarding.

Is there anything process wise that we can do to make refactoring easier for contributors?

Sorry for the miscommunication, though,

Thanks,
Stefan

#20 lore raw

Hi,

Quoted 10 lines · Stefan Beller
Quoted 3 lines · Junio C Hamano

Wait a minute. Is that topic ever shown to work well together with other topics in flight and are now ready to be built upon? I had an impression that it is just starting to get serious reviews.

And I had the impression the serious reviews were done and fine; the only issue would be demonstrating its working fine with other series, where I was also worrying about conflicts with brians series. And to address that, I'd just send series in small sizes.

Some of the patches looked cooked to me and others still do not look cooked yet. I marked the former with Reviewed-by. In general, a few things can help to make the process easier for me:

1. Giving a quick reply to a review to say how the comments were resolved, sometimes even with a resend of that one patch to illustrate. That way the conversation can continue and the individual patch can get to a reviewed state faster, without having to chase between different rerolls of the entire series.

This also has an effect of making the review process more
collaborative: perhaps after seeing how you address their comments, a reviewer may have another idea that they suggest via a patch to squash in, etc.

2. In a reroll, summarizing the result of previous reviews by including acks as appropriate and Reviewed-by if a reviewer granted it. This helps with reviewing the reroll since it tells people where to focus their attention.

[...]

Quoted 2 lines

Is there anything that a contributor can help with that eases refactoring series in flight?

For helping reviewers, see above.

For helping Junio, what I've seen people occasionally do is to locally run a "trial merge" against next and pu and see what semantic or lexical conflicts arise. In the cover letter you can describe these and give Junio advice to make applying the patch easier for him.

[...]

Quoted 1 line

Sorry for the miscommunication, though,

FWIW, even though part 1 doesn't look done to me yet, it looks *close* to done, and I was happy to see the sneak peek at part 2.

Thanks,
Jonathan

#21 lore raw
In reply to Junio C Hamano
Quoted 10 lines · Junio C Hamano
Quoted 2 lines · Stefan Beller

This applies on top of origin/sb/object-store and is the continuation of that series, adding the repository as a context argument to functions.

Wait a minute. Is that topic ever shown to work well together with other topics in flight and are now ready to be built upon? I had an impression that it is just starting to get serious reviews.

Sorry, but I am behind ;-)

OK, so I finally picked up the last round, which wasn't even in my private build. I had the previous round but hadn't convinced myself that my conflict resolution with other topics in flight that were still mushy was correct, so it was out of 'pu', but at least it was in my tree, so that is where my impression came from.

I saw that the way a list-head in a repository is initialized has been revamped in v4, which looked sensible. Will merge it to 'pu' so that I can pick up the ignore-env removal from Duy.

#22 lore raw
Quoted 9 lines · Junio C Hamano

OK, so I finally picked up the last round, which wasn't even in my private build. I had the previous round but hadn't convinced myself that my conflict resolution with other topics in flight that were still mushy was correct, so it was out of 'pu', but at least it was in my tree, so that is where my impression came from.

I saw that the way a list-head in a repository is initialized has been revamped in v4, which looked sensible. Will merge it to 'pu' so that I can pick up the ignore-env removal from Duy.

Thanks,
Stefan