packfile: avoid using the 'the_repository' global variable
Quoted 13 lines · Taylor Blau
Quoted 8 lines · Taylor Blau
Hi Karthik,
Quoted 1 line · Karthik Nayak
Range-diff against v3:
Skimming the range-diff, this new version looks good to me. It would be nice to hear from another reviewer or two before we start merging it down, but I think that this is looking good to me.
Hmmph. I spoke too soon, this new version appears to break CI on Windows, and thus broke the builds of 'jch' (and 'seen', by extension).
Quoted 7 lines
https://github.com/ttaylorr/git/actions/runs/11602969593/job/32309061019Can you have a look?
In the meantime, I'm going to move this out of 'jch' to let CI run there again.
Thanks for letting me know, I think the fix is simply
packfile.c +2 -2
@@ -27,8 +27,8 @@#include "pack-objects.h" struct packfile_config { -unsigned long packed_git_window_size; -unsigned long packed_git_limit; +unsigned long long packed_git_window_size; +unsigned long long packed_git_limit; };#define PACKFILE_CONFIG_INIT \
Tested it on GitLab's CI too this time.
https://gitlab.com/gitlab-org/git/-/jobs/8248707713
Will send in a new version including the fix tomorrow!
Karthik
Quoted 7 lines · karthik nayak
Thanks for letting me know, I think the fix is simply
packfile.c +0
-0
packfile.c line 34
Quoted 10 lines
packfile.c +2
-2@@ -27,8 +27,8 @@#include "pack-objects.h" struct packfile_config { -unsigned long packed_git_window_size; -unsigned long packed_git_limit; +unsigned long long packed_git_window_size; +unsigned long long packed_git_limit; };#define PACKFILE_CONFIG_INIT \
Wait, why did these change from "size_t" to "unsigned long" in your series in the first place? If the goal is moving them into a struct, they should otherwise retain the same types, no?
Two asides, one for your series and one #leftoverbits:
1. Since these are now fields of packfile_config, do they need the long packed_git_ prefix anymore? Just window_size and limit would be nicer, I'd think.
2. I can imagine you might have used "unsigned long" because they are parsed with git_config_ulong(). That is OK on Linux, where size_t and "unsigned long" are the same size (either 32- or 64-bits). But on Windows I think it means that you cannot configure a window larger than 4GB on a 64-bit system. Or ironically you cannot set a total limit larger than 4GB, even though the default is 32TB. ;)
-Peff
Quoted 24 lines · Jeff King
Quoted 7 lines · karthik nayak
Thanks for letting me know, I think the fix is simply
packfile.c +0
-0packfile.c line 34
Quoted 10 lines
packfile.c +2
-2@@ -27,8 +27,8 @@#include "pack-objects.h" struct packfile_config { -unsigned long packed_git_window_size; -unsigned long packed_git_limit; +unsigned long long packed_git_window_size; +unsigned long long packed_git_limit; };#define PACKFILE_CONFIG_INIT \Wait, why did these change from "
size_t" to "unsigned long" in your series in the first place? If the goal is moving them into a struct, they should otherwise retain the same types, no?
Quoted 6 lines
Two asides, one for your series and one #leftoverbits:
1. Since these are now fields of
packfile_config, do they need the longpacked_git_prefix anymore? Justwindow_sizeand limit would be nicer, I'd think.
Moving them to repo_settings means we'd have to keep the long names, otherwise, I agree the shorter names would be better.
Quoted 6 lines
2. I can imagine you might have used "unsigned long" because they are parsed with
git_config_ulong(). That is OK on Linux, wheresize_tand "unsigned long" are the same size (either 32- or 64-bits). But on Windows I think it means that you cannot configure a window larger than 4GB on a 64-bit system. Or ironically you cannot set a total limit larger than 4GB, even though the default is 32TB. ;)
Yup that's the reason I changed them. TIL about size_t and how it works. Thanks, I'll change the types accordingly and push a new version soon.
Quoted 2 lines
-Peff
Quoted 9 lines · karthik nayak
Quoted 6 lines
2. I can imagine you might have used "unsigned long" because they are parsed with
git_config_ulong(). That is OK on Linux, wheresize_tand "unsigned long" are the same size (either 32- or 64-bits). But on Windows I think it means that you cannot configure a window larger than 4GB on a 64-bit system. Or ironically you cannot set a total limit larger than 4GB, even though the default is 32TB. ;)Yup that's the reason I changed them. TIL about
size_tand how it works. Thanks, I'll change the types accordingly and push a new version soon.
Thanks. I think I got so busy talking about the issue that I forgot to mention why I think this is potential #leftoverbits: we probably ought to be parsing that config with git_config_ssize_t() or similar. But that is outside the scope of your series.
-Peff
v5 cover packfile: avoid using the 'the_repository' global variable
In reply to Karthik NayakThe packfile.c file uses the global variable 'the_repository' extensively throughout the code. Let's remove all usecases of this, by modifying the required functions to accept a 'struct repository' instead. This is to clean up usage of global state.
The first 3 patches are mostly internal to packfile.c, we add the repository field to the packed_git struct and this is used to clear up some useages of the global variables.
The next 3 patches are more disruptive, they modify the function definition of odb_pack_name, has_object[_kept]_pack and for_each_packed_object to receive a repository, helping remove other usages of 'the_repository' variable.
Finally, the last two patches deal with global config values. These values are localized.
For v5, I've rebased the series off the new master: 8f8d6eee53 (The seventh batch, 2024-11-01), as a dependency for this series 'jk/dumb-http-finalize' was merged to master. I've found no conflicts while merging with seen & next. But since this series does touch multiple files, there could be future conflicts.
Karthik Nayak (9):
packfile: add repository to struct packed_git
packfile: use repository from packed_git directly
packfile: pass repository to static function in the file
packfile: pass down repository to odb_pack_name
packfile: pass down repository to has_object[_kept]_pack
packfile: pass down repository to for_each_packed_object
config: make delta_base_cache_limit a non-global variable
config: make packed_git_(limit|window_size) non-global variables
midx: add repository to multi_pack_index struct
builtin/cat-file.c | 7 +- builtin/count-objects.c | 2 +- builtin/fast-import.c | 15 ++-- builtin/fsck.c | 20 +++--- builtin/gc.c | 5 +- builtin/index-pack.c | 20 ++++-- builtin/pack-objects.c | 11 +-- builtin/pack-redundant.c | 2 +- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 4 +- config.c | 22 ------ connected.c | 3 +- diff.c | 3 +- environment.c | 3 - environment.h | 1 - fsck.c | 2 +- http.c | 4 +- list-objects.c | 7 +- midx-write.c | 2 +- midx.c | 3 +- midx.h | 3 + object-store-ll.h | 9 ++- pack-bitmap.c | 90 ++++++++++++++---------- pack-objects.h | 3 +- pack-write.c | 1 + pack.h | 1 + packfile.c | 144 ++++++++++++++++++++++----------------- packfile.h | 18 +++-- promisor-remote.c | 2 +- prune-packed.c | 2 +- reachable.c | 4 +- repo-settings.c | 14 ++++ repo-settings.h | 5 ++ revision.c | 13 ++-- tag.c | 2 +- 36 files changed, 261 insertions(+), 190 deletions(-)
Range-diff against v4:
1: b3d518e998 = 1: 6c00e25c86 packfile: add repository to struct `packed_git` 2: bb9d9aa744 = 2: 70fc8a79af packfile: use `repository` from `packed_git` directly 3: d5df50fa36 = 3: 167a1f3a11 packfile: pass `repository` to static function in the file 4: 0107801c3b = 4: b7cfe78217 packfile: pass down repository to `odb_pack_name` 5: 2d7608a367 = 5: 5566f5554c packfile: pass down repository to `has_object[_kept]_pack` 6: 2c84026d02 = 6: 1b26e45a9b packfile: pass down repository to `for_each_packed_object` 7: 84b89c8a0e ! 7: 7654bf5e7e config: make `delta_base_cache_limit` a non-global variable
@@ Commit message
this value from the repository config, since the value is only used once in the entire subsystem.
+ The type of the value is changed from size_t to an unsigned long
+ since the default value is small enough to fit inside the latter on all
+ platforms.
+
These changes are made to remove the usage of delta_base_cache_limit
as a global variable in packfile.c. This brings us one step closer to removing the USE_THE_REPOSITORY_VARIABLE definition in packfile.c
8: 5bbdc7124d ! 8: 2730aacd8e config: make packed_git_(limit|window_size) non-global variables
@@ packfile.c
#include "git-compat-util.h"
#include "environment.h"
-@@
- #include "config.h"
- #include "pack-objects.h"
-
-+struct packfile_config {
-+ unsigned long packed_git_window_size;
-+ unsigned long packed_git_limit;
-+};
-+
-+#define PACKFILE_CONFIG_INIT \
-+{ \
-+ .packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE, \
-+ .packed_git_limit = DEFAULT_PACKED_GIT_LIMIT, \
-+}
-+
- char *odb_pack_name(struct repository *r, struct strbuf *buf,
- const unsigned char *hash, const char *ext)
- {
@@ packfile.c: static size_t pack_mapped;
#define SZ_FMT PRIuMAX
static inline uintmax_t sz_fmt(size_t s) { return s; }
-void pack_report(void)
-+static int packfile_config(const char *var, const char *value,
-+ const struct config_context *ctx, void *cb)
-+{
-+ struct packfile_config *config = cb;
-+
-+ if (!strcmp(var, "core.packedgitwindowsize")) {
-+ int pgsz_x2 = getpagesize() * 2;
-+ config->packed_git_window_size = git_config_ulong(var, value, ctx->kvi);
-+
-+ /* This value must be multiple of (pagesize * 2) */
-+ config->packed_git_window_size /= pgsz_x2;
-+ if (config->packed_git_window_size < 1)
-+ config->packed_git_window_size = 1;
-+ config->packed_git_window_size *= pgsz_x2;
-+ return 0;
-+ } else if (!strcmp(var, "core.packedgitlimit")) {
-+ config->packed_git_limit = git_config_ulong(var, value, ctx->kvi);
-+ return 0;
-+ } else {
-+ return git_default_config(var, value, ctx, cb);
-+ }
-+}
-+
+void pack_report(struct repository *repo)
{
-+ struct packfile_config config = PACKFILE_CONFIG_INIT;
-+ repo_config(repo, packfile_config, &config);
-+
fprintf(stderr,
"pack_report: getpagesize() = %10" SZ_FMT "\n"
"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n"
@@ packfile.c: static size_t pack_mapped;
sz_fmt(getpagesize()),
- sz_fmt(packed_git_window_size),
- sz_fmt(packed_git_limit));
-+ sz_fmt(config.packed_git_window_size),
-+ sz_fmt(config.packed_git_limit));
++ sz_fmt(repo->settings.packed_git_window_size),
++ sz_fmt(repo->settings.packed_git_limit));
fprintf(stderr,
"pack_report: pack_used_ctr = %10u\n"
"pack_report: pack_mmap_calls = %10u\n"
@@ packfile.c: unsigned char *use_pack(struct packed_git *p,
}
if (!win) {
- size_t window_align = packed_git_window_size / 2;
-+ struct packfile_config config = PACKFILE_CONFIG_INIT;
+ size_t window_align;
off_t len;
-
-+ repo_config(p->repo, packfile_config, &config);
-+ window_align = config.packed_git_window_size / 2;
++ struct repo_settings *settings = &p->repo->settings;
+
++ window_align = settings->packed_git_window_size / 2;
+
if (p->pack_fd == -1 && open_packed_git(p))
die("packfile %s cannot be accessed", p->pack_name);
-
+@@ packfile.c: unsigned char *use_pack(struct packed_git *p,
CALLOC_ARRAY(win, 1);
win->offset = (offset / window_align) * window_align;
len = p->pack_size - win->offset;
- if (len > packed_git_window_size)
- len = packed_git_window_size;
-+ if (len > config.packed_git_window_size)
-+ len = config.packed_git_window_size;
++ if (len > settings->packed_git_window_size)
++ len = settings->packed_git_window_size;
win->len = (size_t)len;
pack_mapped += win->len;
- while (packed_git_limit < pack_mapped
+
-+ while (config.packed_git_limit < pack_mapped
++ while (settings->packed_git_limit < pack_mapped
&& unuse_one_window(p))
; /* nothing */
win->base = xmmap_gently(NULL, win->len,
@@ packfile.h: unsigned long repo_approximate_object_count(struct repository *r);
/*
* mmap the index file for the specified packfile (if it is not
+
+ ## repo-settings.c ##
+@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
+ const char *strval;
+ int manyfiles;
+ int read_changed_paths;
++ unsigned long longval;
+
+ if (!r->gitdir)
+ BUG("Cannot add settings for uninitialized repository");
+@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
+ * removed.
+ */
+ r->settings.command_requires_full_index = 1;
++
++ if (!repo_config_get_ulong(r, "core.packedgitwindowsize", &longval)) {
++ int pgsz_x2 = getpagesize() * 2;
++
++ /* This value must be multiple of (pagesize * 2) */
++ longval /= pgsz_x2;
++ if (longval < 1)
++ longval = 1;
++ r->settings.packed_git_window_size = longval * pgsz_x2;
++ }
++
++ if (!repo_config_get_ulong(r, "core.packedgitlimit", &longval))
++ r->settings.packed_git_limit = longval;
+ }
+
+ enum log_refs_config repo_settings_get_log_all_ref_updates(struct repository *repo)
+
+ ## repo-settings.h ##
+@@ repo-settings.h: struct repo_settings {
+
+ int core_multi_pack_index;
+ int warn_ambiguous_refs; /* lazily loaded via accessor */
++
++ size_t packed_git_window_size;
++ size_t packed_git_limit;
+ };
+ #define REPO_SETTINGS_INIT { \
+ .index_version = -1, \
+ .core_untracked_cache = UNTRACKED_CACHE_KEEP, \
+ .fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \
+ .warn_ambiguous_refs = -1, \
++ .packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE, \
++ .packed_git_limit = DEFAULT_PACKED_GIT_LIMIT, \
+ }
+
+ void prepare_repo_settings(struct repository *r); 9: bb15a0be56 = 9: 8e33d40077 midx: add repository to multi_pack_index struct
Signature · 1 line
2.47.0
PATCH v5 1/9 packfile: add repository to struct `packed_git`
The struct packed_git holds information regarding a packed object file. Let's add the repository variable to this object, to represent the repository that this packfile belongs to. This helps remove dependency on the global the_repository object in packfile.c by simply using repository information now readily available in the struct.
We do need to consider that a pack file could be part of the alternates of a repository, but considering that we only have one repository struct and also that we currently anyways use 'the_repository'. We should be OK with this change.
We also modify alloc_packed_git to ensure that the repository is added to newly created packed_git structs. This requires modifying the function and all its callee to pass the repository object down the levels.
--- builtin/fast-import.c | 3 ++- builtin/index-pack.c | 6 ++++-- commit-graph.c | 2 +- connected.c | 3 ++- http.c | 2 +- midx-write.c | 2 +- midx.c | 2 +- object-store-ll.h | 5 +++++ packfile.c | 15 +++++++++------ packfile.h | 6 ++++-- 10 files changed, 30 insertions(+), 16 deletions(-)
builtin/fast-import.c +2 -1
@@ -765,6 +765,7 @@ static void start_packfile(void) p->pack_fd = pack_fd; p->do_not_close =1 ; + p->repo = the_repository; pack_file =hashfd (pack_fd, p->pack_name); pack_data = p;
@@ -888,7 +889,7 @@ static void end_packfile(void) idx_name =keep_pack (create_index ());/* Register the packfile with core git's machinery. */ - new_p =add_packed_git (idx_name,strlen (idx_name),1 ); + new_p =add_packed_git (pack_data->repo, idx_name,strlen (idx_name),1 );if (!new_p)die ("core git rejected index %s" , idx_name); all_packs[pack_id] = new_p;
builtin/index-pack.c +4 -2
@@ -1552,7 +1552,8 @@ static void final(const char *final_pack_name, const char *curr_pack_name,if (do_fsck_object) {struct packed_git *p; - p =add_packed_git (final_index_name,strlen (final_index_name),0 ); + p =add_packed_git (the_repository, final_index_name, +strlen (final_index_name),0 );if (p)install_packed_git (the_repository, p); }
@@ -1650,7 +1651,8 @@ static void read_v2_anomalous_offsets(struct packed_git *p,static void read_idx_option (struct pack_idx_option *opts,const char *pack_name) { -struct packed_git *p =add_packed_git (pack_name,strlen (pack_name),1 ); +struct packed_git *p =add_packed_git (the_repository, pack_name, +strlen (pack_name),1 );if (!p)die (_ ("Cannot open existing pack file '%s'" ), pack_name);
commit-graph.c +1 -1
@@ -1914,7 +1914,7 @@ static int fill_oids_from_packs(struct write_commit_graph_context *ctx,struct packed_git *p;strbuf_setlen (&packname, dirlen);strbuf_addstr (&packname, pack_indexes->items[i].string); - p =add_packed_git (packname.buf, packname.len,1 ); + p =add_packed_git (ctx->r, packname.buf, packname.len,1 );if (!p) { ret =error (_ ("error adding pack %s" ), packname.buf); goto cleanup;
connected.c +2 -1
@@ -54,7 +54,8 @@ int check_connected(oid_iterate_fn fn, void *cb_data,strbuf_add (&idx_file, transport->pack_lockfiles.items[0 ].string, base_len);strbuf_addstr (&idx_file,".idx" ); - new_pack =add_packed_git (idx_file.buf, idx_file.len,1 ); + new_pack =add_packed_git (the_repository, idx_file.buf, + idx_file.len,1 );strbuf_release (&idx_file); }
http.c +1 -1
@@ -2439,7 +2439,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,if (!tmp_idx)return -1 ; - new_pack =parse_pack_index (sha1, tmp_idx); + new_pack =parse_pack_index (the_repository, sha1, tmp_idx);if (!new_pack) {unlink (tmp_idx);free (tmp_idx);
midx-write.c +1 -1
@@ -154,7 +154,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,return ;ALLOC_GROW (ctx->info, ctx->nr +1 , ctx->alloc); - p =add_packed_git (full_path, full_path_len,0 ); + p =add_packed_git (the_repository, full_path, full_path_len,0 );if (!p) {warning (_ ("failed to add packfile '%s'" ), full_path);
midx.c +1 -1
@@ -464,7 +464,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,strhash (key.buf), key.buf, struct packed_git, packmap_ent);if (!p) { - p =add_packed_git (pack_name.buf, pack_name.len, m->local); + p =add_packed_git (r, pack_name.buf, pack_name.len, m->local);if (p) {install_packed_git (r, p);list_add_tail (&p->mru, &r->objects->packed_git_mru);
object-store-ll.h +5 -0
@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;struct object_directory {struct object_directory *next;
@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo dentoes the repository this packed file belongs to */ +struct repository *repo; +/* something like ".git/objects/pack/xxxxx.pack" */ char pack_name[FLEX_ARRAY ];/* more */ };
packfile.c +9 -6
@@ -217,11 +217,12 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)return ntohl (level1_ofs[value]); } -static struct packed_git *alloc_packed_git (int extra) +static struct packed_git *alloc_packed_git (struct repository *r,int extra) {struct packed_git *p =xmalloc (st_add (sizeof (*p), extra));memset (p,0 ,sizeof (*p)); p->pack_fd =-1 ; + p->repo = r;return p; }
@@ -233,11 +234,12 @@ static char *pack_path_from_idx(const char *idx_path)return xstrfmt ("%.*s.pack" , (int )len, idx_path); } -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path) {char *path =pack_path_from_idx (idx_path);size_t alloc =st_add (strlen (path),1 ); -struct packed_git *p =alloc_packed_git (alloc); +struct packed_git *p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path);
@@ -703,7 +705,8 @@ void unuse_pack(struct pack_window **w_cursor) } } -struct packed_git *add_packed_git (const char *path,size_t path_len,int local) +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local) {struct stat st;size_t alloc;
@@ -721,7 +724,7 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)* the use xsnprintf double-checks that) */ alloc =st_add3 (path_len,strlen (".promisor" ),1 ); - p =alloc_packed_git (alloc); + p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, path_len);xsnprintf (p->pack_name + path_len, alloc - path_len,".keep" );
@@ -877,7 +880,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,/* Don't reopen a pack we already have. */ if (!hashmap_get (&data->r->objects->pack_map, &hent, pack_name)) { - p =add_packed_git (full_name, full_name_len, data->local); + p =add_packed_git (data->r, full_name, full_name_len, data->local);if (p)install_packed_git (data->r, p); }
packfile.h +4 -2
@@ -46,7 +46,8 @@ const char *pack_basename(struct packed_git *p);* and does not add the resulting packed_git struct to the internal list of * packs. You probably want add_packed_git() instead. */ -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path);typedef void each_file_in_pack_dir_fn (const char *full_path,size_t full_path_len,const char *file_name,void *data);
@@ -113,7 +114,8 @@ void close_pack(struct packed_git *);void close_object_store (struct raw_object_store *o);void unuse_pack (struct pack_window **);void clear_delta_base_cache (void ); -struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local);/* * Unlink the .pack and associated extension files.
Signature · 1 line
2.47.0
PATCH v5 2/9 packfile: use `repository` from `packed_git` directly
In reply to Karthik NayakIn the previous commit, we introduced the repository structure inside packed_git. This provides an alternative route instead of using the global the_repository variable. Let's modify packfile.c now to use this field wherever possible instead of relying on the global state. There are still a few instances of the_repository usage in the file, where there is no struct packed_git locally available, which will be fixed in the following commits.
--- packfile.c | 50 +++++++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 23 deletions(-)
packfile.c +27 -23
@@ -79,7 +79,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)size_t idx_size;int fd =git_open (path), ret;struct stat st; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (fd <0 )return -1 ;
@@ -243,7 +243,7 @@ struct packed_git *parse_pack_index(struct repository *r, unsigned char *sha1,memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path); -hashcpy (p->hash, sha1, the_repository->hash_algo); +hashcpy (p->hash, sha1, p->repo->hash_algo);if (check_packed_git_idx (idx_path, p)) {free (p);return NULL ;
@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = the_repository->objects->packed_git; p; p = p->next) +for (p = current->repo->objects->packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -540,7 +540,7 @@ static int open_packed_git_1(struct packed_git *p)unsigned char hash[GIT_MAX_RAWSZ ];unsigned char *idx_hash;ssize_t read_result; -const unsigned hashsz = the_hash_algo->rawsz; +const unsigned hashsz = p->repo->hash_algo->rawsz;if (open_pack_index (p))return error ("packfile %s index unavailable" , p->pack_name);
@@ -597,7 +597,7 @@ static int open_packed_git_1(struct packed_git *p)if (read_result != hashsz)return error ("packfile %s signature is unavailable" , p->pack_name); idx_hash = ((unsigned char *)p->index_data) + p->index_size - hashsz *2 ; -if (!hasheq (hash, idx_hash, the_repository->hash_algo)) +if (!hasheq (hash, idx_hash, p->repo->hash_algo))return error ("packfile %s does not match index" , p->pack_name);return 0 ; }
@@ -637,7 +637,7 @@ unsigned char *use_pack(struct packed_git *p,*/ if (!p->pack_size && p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name); -if (offset > (p->pack_size - the_hash_algo->rawsz)) +if (offset > (p->pack_size - p->repo->hash_algo->rawsz))die ("offset beyond end of packfile (truncated pack?)" );if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" ));
@@ -711,6 +711,7 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,struct stat st;size_t alloc;struct packed_git *p; +struct object_id oid;/* * Make sure a corresponding .pack file exists and that
@@ -751,9 +752,13 @@ struct packed_git *add_packed_git(struct repository *r, const char *path, p->pack_size = st.st_size; p->pack_local = local; p->mtime = st.st_mtime; -if (path_len < the_hash_algo->hexsz || -get_hash_hex (path + path_len - the_hash_algo->hexsz, p->hash)) -hashclr (p->hash, the_repository->hash_algo); +if (path_len < r->hash_algo->hexsz || +get_oid_hex_algop (path + path_len - r->hash_algo->hexsz, &oid, + r->hash_algo)) +hashclr (p->hash, r->hash_algo); +else +hashcpy (p->hash, oid.hash, r->hash_algo); +return p; }
@@ -1243,9 +1248,9 @@ off_t get_delta_base(struct packed_git *p, }else if (type ==OBJ_REF_DELTA ) {/* The base entry _must_ be in the same pack */ struct object_id oid; -oidread (&oid, base_info, the_repository->hash_algo); +oidread (&oid, base_info, p->repo->hash_algo); base_offset =find_pack_entry_one (&oid, p); - *curpos += the_hash_algo->rawsz; + *curpos += p->repo->hash_algo->rawsz; }else die ("I am totally screwed" );return base_offset;
@@ -1266,7 +1271,7 @@ static int get_delta_base_oid(struct packed_git *p, {if (type ==OBJ_REF_DELTA ) {unsigned char *base =use_pack (p, w_curs, curpos,NULL ); -oidread (oid, base, the_repository->hash_algo); +oidread (oid, base, p->repo->hash_algo);return 0 ; }else if (type ==OBJ_OFS_DELTA ) {uint32_t base_pos;
@@ -1608,7 +1613,7 @@ int packed_object_info(struct repository *r, struct packed_git *p, goto out; } }else -oidclr (oi->delta_base_oid, the_repository->hash_algo); +oidclr (oi->delta_base_oid, p->repo->hash_algo); } oi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :
@@ -1897,7 +1902,7 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32 {const unsigned char *index_fanout = p->index_data;const unsigned char *index_lookup; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;int index_lookup_width;if (!index_fanout)
@@ -1922,7 +1927,7 @@ int nth_packed_object_id(struct object_id *oid, uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (!index) {if (open_pack_index (p))return -1 ;
@@ -1933,11 +1938,10 @@ int nth_packed_object_id(struct object_id *oid, index +=4 *256 ;if (p->index_version ==1 ) {oidread (oid, index +st_add (st_mult (hashsz +4 , n),4 ), - the_repository->hash_algo); + p->repo->hash_algo); }else { index +=8 ; -oidread (oid, index +st_mult (hashsz, n), - the_repository->hash_algo); +oidread (oid, index +st_mult (hashsz, n), p->repo->hash_algo); }return 0 ; }
@@ -1959,7 +1963,7 @@ void check_pack_index_ptr(const struct packed_git *p, const void *vptr)off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz; index +=4 *256 ;if (p->index_version ==1 ) {return ntohl (*((uint32_t *)(index +st_mult (hashsz +4 , n))));
@@ -2159,7 +2163,7 @@ int for_each_object_in_pack(struct packed_git *p,int r =0 ;if (flags &FOR_EACH_OBJECT_PACK_ORDER ) { -if (load_pack_revindex (the_repository, p)) +if (load_pack_revindex (p->repo, p))return -1 ; }
@@ -2227,7 +2231,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, }static int add_promisor_object (const struct object_id *oid, - struct packed_git *pack UNUSED, +struct packed_git *pack, uint32_t pos UNUSED,void *set_) {
@@ -2235,12 +2239,12 @@ static int add_promisor_object(const struct object_id *oid,struct object *obj;int we_parsed_object; - obj =lookup_object (the_repository, oid); + obj =lookup_object (pack->repo, oid);if (obj && obj->parsed) { we_parsed_object =0 ; }else { we_parsed_object =1 ; - obj =parse_object (the_repository, oid); + obj =parse_object (pack->repo, oid); }if (!obj)
Signature · 1 line
2.47.0
PATCH v5 3/9 packfile: pass `repository` to static function in the file
In reply to Karthik NayakSome of the static functions in the packfile.c access global variables, which can simply be avoiding by passing the repository struct down to them. Let's do that.
--- packfile.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
packfile.c +8 -7
@@ -460,13 +460,13 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc *accept_windows_inuse = has_windows_inuse; } -static int close_one_pack (void ) +static int close_one_pack (struct repository *r) {struct packed_git *p, *lru_p =NULL ;struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = the_repository->objects->packed_git; p; p = p->next) { +for (p = r->objects->packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -555,7 +555,7 @@ static int open_packed_git_1(struct packed_git *p) pack_max_fds =1 ; } -while (pack_max_fds <= pack_open_fds &&close_one_pack ()) +while (pack_max_fds <= pack_open_fds &&close_one_pack (p->repo)) ;/* nothing */ p->pack_fd =git_open (p->pack_name);
@@ -610,7 +610,8 @@ static int open_packed_git(struct packed_git *p)return -1 ; } -static int in_window (struct pack_window *win,off_t offset) +static int in_window (struct repository *r,struct pack_window *win, +off_t offset) {/* We must promise at least one full hash after the * offset is available from this window, otherwise the offset
@@ -620,7 +621,7 @@ static int in_window(struct pack_window *win, off_t offset)*/ off_t win_off = win->offset;return win_off <= offset - && (offset + the_hash_algo->rawsz) <= (win_off + win->len); + && (offset + r->hash_algo->rawsz) <= (win_off + win->len); } unsigned char *use_pack(struct packed_git *p,
@@ -642,11 +643,11 @@ unsigned char *use_pack(struct packed_git *p,if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" )); -if (!win || !in_window (win, offset)) { +if (!win || !in_window (p->repo, win, offset)) {if (win) win->inuse_cnt--;for (win = p->windows; win; win = win->next) { -if (in_window (win, offset)) +if (in_window (p->repo, win, offset))break ; }if (!win) {
Signature · 1 line
2.47.0
PATCH v5 4/9 packfile: pass down repository to `odb_pack_name`
In reply to Karthik NayakThe function odb_pack_name currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers.
--- builtin/fast-import.c | 8 ++++---- builtin/index-pack.c | 4 ++-- builtin/pack-redundant.c | 2 +- http.c | 2 +- packfile.c | 9 ++++----- packfile.h | 3 ++- 6 files changed, 14 insertions(+), 14 deletions(-)
builtin/fast-import.c +4 -4
@@ -806,7 +806,7 @@ static char *keep_pack(const char *curr_index_name)struct strbuf name =STRBUF_INIT ;int keep_fd; -odb_pack_name (&name, pack_data->hash,"keep" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"keep" ); keep_fd =odb_pack_keep (name.buf);if (keep_fd <0 )die_errno ("cannot create keep file" );
@@ -814,11 +814,11 @@ static char *keep_pack(const char *curr_index_name)if (close (keep_fd))die_errno ("failed to write keep file" ); -odb_pack_name (&name, pack_data->hash,"pack" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"pack" );if (finalize_object_file (pack_data->pack_name, name.buf))die ("cannot store pack file" ); -odb_pack_name (&name, pack_data->hash,"idx" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"idx" );if (finalize_object_file (curr_index_name, name.buf))die ("cannot store index file" );free ((void *)curr_index_name);
@@ -832,7 +832,7 @@ static void unkeep_all_packs(void)for (k =0 ; k < pack_id; k++) {struct packed_git *p = all_packs[k]; -odb_pack_name (&name, p->hash,"keep" ); +odb_pack_name (p->repo, &name, p->hash,"keep" );unlink_or_warn (name.buf); }strbuf_release (&name);
builtin/index-pack.c +2 -2
@@ -1479,7 +1479,7 @@ static void write_special_file(const char *suffix, const char *msg,if (pack_name) filename =derive_filename (pack_name,"pack" , suffix, &name_buf);else - filename =odb_pack_name (&name_buf, hash, suffix); + filename =odb_pack_name (the_repository, &name_buf, hash, suffix); fd =odb_pack_keep (filename);if (fd <0 ) {
@@ -1507,7 +1507,7 @@ static void rename_tmp_packfile(const char **final_name, {if (!*final_name ||strcmp (*final_name, curr_name)) {if (!*final_name) - *final_name =odb_pack_name (name, hash, ext); + *final_name =odb_pack_name (the_repository, name, hash, ext);if (finalize_object_file (curr_name, *final_name))die (_ ("unable to rename temporary '*.%s' file to '%s'" ), ext, *final_name);
builtin/pack-redundant.c +1 -1
@@ -690,7 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix UNUSED, s pl = red =pack_list_difference (local_packs, min);while (pl) {printf ("%s\n%s\n" , -odb_pack_name (&idx_name, pl->pack->hash,"idx" ), +odb_pack_name (pl->pack->repo, &idx_name, pl->pack->hash,"idx" ), pl->pack->pack_name); pl = pl->next; }
http.c +1 -1
@@ -2581,7 +2581,7 @@ struct http_pack_request *new_direct_http_pack_request( preq->url = url; -odb_pack_name (&preq->tmpfile, packed_git_hash,"pack" ); +odb_pack_name (the_repository, &preq->tmpfile, packed_git_hash,"pack" );strbuf_addstr (&preq->tmpfile,".temp" ); preq->packfile =fopen (preq->tmpfile.buf,"a" );if (!preq->packfile) {
packfile.c +4 -5
@@ -25,13 +25,12 @@#include "pack-revindex.h" #include "promisor-remote.h" -char *odb_pack_name (struct strbuf *buf, -const unsigned char *hash, -const char *ext) +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext) {strbuf_reset (buf); -strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (the_repository), -hash_to_hex (hash), ext); +strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (r), +hash_to_hex_algop (hash, r->hash_algo), ext);return buf->buf; }
packfile.h +2 -1
@@ -29,7 +29,8 @@ struct pack_entry {* * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" */ -char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext);/* * Return the basename of the packfile, omitting any containing directory
Signature · 1 line
2.47.0
PATCH v5 5/9 packfile: pass down repository to `has_object[_kept]_pack`
In reply to Karthik NayakThe functions has_object[_kept]_pack currently rely on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from these functions and any related ones.
--- builtin/count-objects.c | 2 +- builtin/fsck.c | 2 +- builtin/pack-objects.c | 4 ++-- diff.c | 3 ++- list-objects.c | 3 ++- pack-bitmap.c | 2 +- packfile.c | 9 +++++---- packfile.h | 5 +++-- prune-packed.c | 2 +- reachable.c | 2 +- revision.c | 4 ++-- 11 files changed, 21 insertions(+), 17 deletions(-)
builtin/count-objects.c +1 -1
@@ -67,7 +67,7 @@ static int count_loose(const struct object_id *oid, const char *path,else { loose_size +=on_disk_bytes (st); loose++; -if (verbose &&has_object_pack (oid)) +if (verbose &&has_object_pack (the_repository, oid)) packed_loose++; }return 0 ;
builtin/fsck.c +1 -1
@@ -272,7 +272,7 @@ static void check_reachable_object(struct object *obj)if (!(obj->flags &HAS_OBJ )) {if (is_promisor_object (&obj->oid))return ; -if (has_object_pack (&obj->oid)) +if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */ printf_ln(_("missing %s %s"), printable_type(&obj->oid, obj->type),
builtin/pack-objects.c +2 -2
@@ -1529,7 +1529,7 @@ static int want_found_object(const struct object_id *oid, int exclude,return 0 ;if (ignore_packed_keep_in_core && p->pack_keep_in_core)return 0 ; -if (has_object_kept_pack (oid, flags)) +if (has_object_kept_pack (p->repo, oid, flags))return 0 ; }
@@ -3627,7 +3627,7 @@ static void show_cruft_commit(struct commit *commit, void *data) static int cruft_include_check_obj(struct object *obj, void *data UNUSED) { -return !has_object_kept_pack (&obj->oid,IN_CORE_KEEP_PACKS ); +return !has_object_kept_pack (to_pack.repo, &obj->oid,IN_CORE_KEEP_PACKS ); }static int cruft_include_check (struct commit *commit,void *data)
diff.c +2 -1
@@ -4041,7 +4041,8 @@ static int reuse_worktree_file(struct index_state *istate,* objects however would tend to be slower as they need * to be individually opened and inflated. */ -if (!FAST_WORKING_DIRECTORY && !want_file &&has_object_pack (oid)) +if (!FAST_WORKING_DIRECTORY && !want_file && +has_object_pack (istate->repo, oid))return 0 ;/*
list-objects.c +2 -1
@@ -41,7 +41,8 @@ static void show_object(struct traversal_context *ctx, {if (!ctx->show_object)return ; -if (ctx->revs->unpacked &&has_object_pack (&object->oid)) +if (ctx->revs->unpacked &&has_object_pack (ctx->revs->repo, + &object->oid))return ; ctx->show_object (object, name, ctx->show_data);
pack-bitmap.c +1 -1
@@ -1889,7 +1889,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,bitmap_unset (result, i);for (i =0 ; i < eindex->count; ++i) { -if (has_object_pack (&eindex->objects[i]->oid)) +if (has_object_pack (the_repository, &eindex->objects[i]->oid))bitmap_unset (result, objects_nr + i); } }
packfile.c +5 -4
@@ -2143,16 +2143,17 @@ int find_kept_pack_entry(struct repository *r,return 0 ; } -int has_object_pack (const struct object_id *oid) +int has_object_pack (struct repository *r,const struct object_id *oid) {struct pack_entry e; -return find_pack_entry (the_repository, oid, &e); +return find_pack_entry (r, oid, &e); } -int has_object_kept_pack (const struct object_id *oid,unsigned flags) +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags) {struct pack_entry e; -return find_kept_pack_entry (the_repository, oid, flags, &e); +return find_kept_pack_entry (r, oid, flags, &e); } int for_each_object_in_pack(struct packed_git *p,
packfile.h +3 -2
@@ -193,8 +193,9 @@ const struct packed_git *has_packed_and_bad(struct repository *, const struct obint find_pack_entry (struct repository *r,const struct object_id *oid,struct pack_entry *e);int find_kept_pack_entry (struct repository *r,const struct object_id *oid,unsigned flags,struct pack_entry *e); -int has_object_pack (const struct object_id *oid); -int has_object_kept_pack (const struct object_id *oid,unsigned flags); +int has_object_pack (struct repository *r,const struct object_id *oid); +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags);/* * Return 1 if an object in a promisor packfile is or refers to the given
prune-packed.c +1 -1
@@ -24,7 +24,7 @@ static int prune_object(const struct object_id *oid, const char *path, {int *opts = data; -if (!has_object_pack (oid)) +if (!has_object_pack (the_repository, oid))return 0 ;if (*opts &PRUNE_PACKED_DRY_RUN )
reachable.c +1 -1
@@ -239,7 +239,7 @@ static int want_recent_object(struct recent_data *data, const struct object_id *oid) {if (data->ignore_in_core_kept_packs && -has_object_kept_pack (oid,IN_CORE_KEEP_PACKS )) +has_object_kept_pack (data->revs->repo, oid,IN_CORE_KEEP_PACKS ))return 0 ;return 1 ; }
revision.c +2 -2
@@ -4103,10 +4103,10 @@ enum commit_action get_commit_action(struct rev_info *revs, struct commit *commi {if (commit->object.flags &SHOWN )return commit_ignore; -if (revs->unpacked &&has_object_pack (&commit->object.oid)) +if (revs->unpacked &&has_object_pack (revs->repo, &commit->object.oid))return commit_ignore;if (revs->no_kept_objects) { -if (has_object_kept_pack (&commit->object.oid, +if (has_object_kept_pack (revs->repo, &commit->object.oid, revs->keep_pack_cache_flags))return commit_ignore; }
Signature · 1 line
2.47.0
PATCH v5 6/9 packfile: pass down repository to `for_each_packed_object`
In reply to Karthik NayakThe function for_each_packed_object currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from this function and closely related function is_promisor_object.
--- builtin/cat-file.c | 7 ++++--- builtin/fsck.c | 18 +++++++++++------- builtin/pack-objects.c | 7 +++++-- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 2 +- fsck.c | 2 +- list-objects.c | 4 ++-- object-store-ll.h | 4 ++-- packfile.c | 14 +++++++------- packfile.h | 2 +- promisor-remote.c | 2 +- reachable.c | 2 +- revision.c | 9 +++++---- tag.c | 2 +- 15 files changed, 44 insertions(+), 35 deletions(-)
builtin/cat-file.c +4 -3
@@ -827,15 +827,16 @@ static int batch_objects(struct batch_options *opt) cb.seen = &seen;for_each_loose_object (batch_unordered_loose, &cb,0 ); -for_each_packed_object (batch_unordered_packed, &cb, -FOR_EACH_OBJECT_PACK_ORDER ); +for_each_packed_object (the_repository, batch_unordered_packed, + &cb,FOR_EACH_OBJECT_PACK_ORDER );oidset_clear (&seen); }else {struct oid_array sa =OID_ARRAY_INIT ;for_each_loose_object (collect_loose_object, &sa,0 ); -for_each_packed_object (collect_packed_object, &sa,0 ); +for_each_packed_object (the_repository, collect_packed_object, + &sa,0 );oid_array_for_each_unique (&sa, batch_object_cb, &cb);
builtin/fsck.c +11 -7
@@ -150,7 +150,7 @@ static int mark_object(struct object *obj, enum object_type type,return 0 ; obj->flags |=REACHABLE ; -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))/* * Further recursion does not need to be performed on this * object since it is a promisor object (so it does not need to
@@ -270,7 +270,7 @@ static void check_reachable_object(struct object *obj)* do a full fsck */ if (!(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */
@@ -391,7 +391,10 @@ static void check_connectivity(void)* traversal. */ for_each_loose_object (mark_loose_unreachable_referents,NULL ,0 ); -for_each_packed_object (mark_packed_unreachable_referents,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_unreachable_referents, +NULL , +0 ); }/* Look up all the requirements, warn about missing objects.. */
@@ -488,7 +491,7 @@ static void fsck_handle_reflog_oid(const char *refname, struct object_id *oid, refname, timestamp); obj->flags |=USED ;mark_object_reachable (obj); - }else if (!is_promisor_object (oid)) { + }else if (!is_promisor_object (the_repository, oid)) {error (_ ("%s: invalid reflog entry %s" ), refname,oid_to_hex (oid)); errors_found |=ERROR_REACHABLE ;
@@ -531,7 +534,7 @@ static int fsck_handle_ref(const char *refname, const char *referent UNUSED, con obj =parse_object (the_repository, oid);if (!obj) { -if (is_promisor_object (oid)) { +if (is_promisor_object (the_repository, oid)) {/* * Increment default_refs anyway, because this is a * valid ref.
@@ -966,7 +969,8 @@ int cmd_fsck(int argc,if (connectivity_only) {for_each_loose_object (mark_loose_for_connectivity,NULL ,0 ); -for_each_packed_object (mark_packed_for_connectivity,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_for_connectivity,NULL ,0 ); }else {prepare_alt_odb (the_repository);for (odb = the_repository->objects->odb; odb; odb = odb->next)
@@ -1011,7 +1015,7 @@ int cmd_fsck(int argc, &oid);if (!obj || !(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&oid)) +if (is_promisor_object (the_repository, &oid))continue ;error (_ ("%s: object missing" ),oid_to_hex (&oid)); errors_found |=ERROR_OBJECT ;
builtin/pack-objects.c +5 -2
@@ -3858,7 +3858,8 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,* Quietly ignore EXPECTED missing objects. This avoids problems with * staging them now and getting an odd error later. */ -if (!has_object (the_repository, &obj->oid,0 ) &&is_promisor_object (&obj->oid)) +if (!has_object (the_repository, &obj->oid,0 ) && +is_promisor_object (to_pack.repo, &obj->oid))return ;show_object (obj, name, data);
@@ -3927,7 +3928,9 @@ static int add_object_in_unpacked_pack(const struct object_id *oid,static void add_objects_in_unpacked_packs (void ) { - if (for_each_packed_object(add_object_in_unpacked_pack, NULL, + if (for_each_packed_object(to_pack.repo, + add_object_in_unpacked_pack, + NULL, FOR_EACH_OBJECT_PACK_ORDER | FOR_EACH_OBJECT_LOCAL_ONLY | FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS |
builtin/repack.c +1 -1
@@ -404,7 +404,7 @@ static void repack_promisor_objects(const struct pack_objects_args *args,* {type -> existing pack order} ordering when computing deltas instead * of a {type -> size} ordering, which may produce better deltas. */ -for_each_packed_object (write_oid, &cmd, +for_each_packed_object (the_repository, write_oid, &cmd,FOR_EACH_OBJECT_PROMISOR_ONLY );if (cmd.in ==-1 ) {
builtin/rev-list.c +1 -1
@@ -121,7 +121,7 @@ static inline void finish_object__ma(struct object *obj)return ;case MA_ALLOW_PROMISOR : -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;die ("unexpected missing %s object '%s'" ,type_name (obj->type),oid_to_hex (&obj->oid));
commit-graph.c +1 -1
@@ -1960,7 +1960,7 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx) ctx->progress =start_delayed_progress (_ ("Finding commits for commit graph among packed objects" ), ctx->approx_nr_objects); -for_each_packed_object (add_packed_commits, ctx, +for_each_packed_object (ctx->r, add_packed_commits, ctx,FOR_EACH_OBJECT_PACK_ORDER );if (ctx->progress_done < ctx->approx_nr_objects)display_progress (ctx->progress, ctx->approx_nr_objects);
fsck.c +1 -1
@@ -1295,7 +1295,7 @@ static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done, buf =repo_read_object_file (the_repository, oid, &type, &size);if (!buf) { -if (is_promisor_object (oid)) +if (is_promisor_object (the_repository, oid))continue ; ret |= report(options, oid, OBJ_BLOB, msg_missing,
list-objects.c +2 -2
@@ -75,7 +75,7 @@ static void process_blob(struct traversal_context *ctx,*/ if (ctx->revs->exclude_promisor_objects && !repo_has_object_file (the_repository, &obj->oid) && -is_promisor_object (&obj->oid)) +is_promisor_object (ctx->revs->repo, &obj->oid))return ; pathlen = path->len;
@@ -180,7 +180,7 @@ static void process_tree(struct traversal_context *ctx,* an incomplete list of missing objects. */ if (revs->exclude_promisor_objects && -is_promisor_object (&obj->oid)) +is_promisor_object (revs->repo, &obj->oid))return ;if (!revs->do_not_die_on_missing_objects)
object-store-ll.h +2 -2
@@ -550,7 +550,7 @@ typedef int each_packed_object_fn(const struct object_id *oid,int for_each_object_in_pack (struct packed_git *p,each_packed_object_fn ,void *data,enum for_each_object_flags flags); -int for_each_packed_object (each_packed_object_fn ,void *, -enum for_each_object_flags flags); +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags);#endif /* OBJECT_STORE_LL_H */
packfile.c +7 -7
@@ -2200,15 +2200,15 @@ int for_each_object_in_pack(struct packed_git *p,return r; } -int for_each_packed_object (each_packed_object_fn cb,void *data, -enum for_each_object_flags flags) +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags) {struct packed_git *p;int r =0 ;int pack_errors =0 ; -prepare_packed_git (the_repository); -for (p =get_all_packs (the_repository); p; p = p->next) { +prepare_packed_git (repo); +for (p =get_all_packs (repo); p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ; if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
@@ -2286,14 +2286,14 @@ static int add_promisor_object(const struct object_id *oid,return 0 ; } -int is_promisor_object (const struct object_id *oid) +int is_promisor_object (struct repository *r,const struct object_id *oid) {static struct oidset promisor_objects;static int promisor_objects_prepared;if (!promisor_objects_prepared) { -if (repo_has_promisor_remote (the_repository)) { -for_each_packed_object (add_promisor_object, +if (repo_has_promisor_remote (r)) { +for_each_packed_object (r, add_promisor_object, &promisor_objects,FOR_EACH_OBJECT_PROMISOR_ONLY |FOR_EACH_OBJECT_PACK_ORDER );
packfile.h +1 -1
@@ -201,7 +201,7 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,* Return 1 if an object in a promisor packfile is or refers to the given * object, 0 otherwise. */ -int is_promisor_object (const struct object_id *oid); +int is_promisor_object (struct repository *r,const struct object_id *oid);/* * Expose a function for fuzz testing.
promisor-remote.c +1 -1
@@ -283,7 +283,7 @@ void promisor_remote_get_direct(struct repository *repo, }for (i =0 ; i < remaining_nr; i++) { -if (is_promisor_object (&remaining_oids[i])) +if (is_promisor_object (repo, &remaining_oids[i]))die (_ ("could not fetch %s from promisor remote" ),oid_to_hex (&remaining_oids[i])); }
reachable.c +1 -1
@@ -324,7 +324,7 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,if (ignore_in_core_kept_packs) flags |=FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS ; - r =for_each_packed_object (add_recent_packed, &data, flags); + r =for_each_packed_object (revs->repo, add_recent_packed, &data, flags); done:oidset_clear (&data.extra_recent_oids);
revision.c +5 -4
@@ -390,7 +390,8 @@ static struct object *get_reference(struct rev_info *revs, const char *name,if (!object) {if (revs->ignore_missing)return NULL ; -if (revs->exclude_promisor_objects &&is_promisor_object (oid)) +if (revs->exclude_promisor_objects && +is_promisor_object (revs->repo, oid))return NULL ;if (revs->do_not_die_on_missing_objects) {oidset_insert (&revs->missing_commits, oid);
@@ -432,7 +433,7 @@ static struct commit *handle_commit(struct rev_info *revs,if (revs->ignore_missing_links || (flags &UNINTERESTING ))return NULL ;if (revs->exclude_promisor_objects && -is_promisor_object (&tag->tagged->oid)) +is_promisor_object (revs->repo, &tag->tagged->oid))return NULL ;if (revs->do_not_die_on_missing_objects && oid) {oidset_insert (&revs->missing_commits, oid);
@@ -1211,7 +1212,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit, revs->do_not_die_on_missing_objects;if (repo_parse_commit_gently (revs->repo, p, gently) <0 ) {if (revs->exclude_promisor_objects && -is_promisor_object (&p->object.oid)) { +is_promisor_object (revs->repo, &p->object.oid)) {if (revs->first_parent_only)break ;continue ;
@@ -3915,7 +3916,7 @@ int prepare_revision_walk(struct rev_info *revs) revs->treesame.name ="treesame" ;if (revs->exclude_promisor_objects) { -for_each_packed_object (mark_uninteresting, revs, +for_each_packed_object (revs->repo, mark_uninteresting, revs,FOR_EACH_OBJECT_PROMISOR_ONLY ); }
tag.c +1 -1
@@ -84,7 +84,7 @@ struct object *deref_tag(struct repository *r, struct object *o, const char *war o =NULL ; }if (!o && warn) { -if (last_oid &&is_promisor_object (last_oid)) +if (last_oid &&is_promisor_object (r, last_oid))return NULL ;if (!warnlen) warnlen =strlen (warn);
Signature · 1 line
2.47.0
PATCH v5 7/9 config: make `delta_base_cache_limit` a non-global variable
In reply to Karthik NayakThe delta_base_cache_limit variable is a global config variable used by multiple subsystems. Let's make this non-global, by adding this variable to the stack of each of the subsystems where it is used.
In gc.c we add it to the gc_config struct and also the constructor function. In index-pack.c we add it to the pack_idx_option struct and its constructor. Finally, in packfile.c we dynamically retrieve this value from the repository config, since the value is only used once in the entire subsystem.
The type of the value is changed from size_t to an unsigned long since the default value is small enough to fit inside the latter on all platforms.
These changes are made to remove the usage of delta_base_cache_limit as a global variable in packfile.c. This brings us one step closer to removing the USE_THE_REPOSITORY_VARIABLE definition in packfile.c which we complete in the next patch.
--- builtin/gc.c | 5 ++++- builtin/index-pack.c | 10 +++++++--- config.c | 5 ----- environment.c | 1 - environment.h | 1 - pack-objects.h | 3 ++- pack-write.c | 1 + pack.h | 1 + packfile.c | 13 +++++++++++-- 9 files changed, 26 insertions(+), 14 deletions(-)
builtin/gc.c +4 -1
@@ -138,6 +138,7 @@ struct gc_config {char *repack_filter_to;unsigned long big_pack_threshold;unsigned long max_delta_cache_size; +unsigned long delta_base_cache_limit; };#define GC_CONFIG_INIT { \
@@ -153,6 +154,7 @@ struct gc_config { .prune_expire = xstrdup("2.weeks.ago"), \ .prune_worktrees_expire = xstrdup("3.months.ago"), \ .max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \ + .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \ }static void gc_config_release (struct gc_config *cfg)
@@ -205,6 +207,7 @@ static void gc_config(struct gc_config *cfg)git_config_get_ulong ("gc.bigpackthreshold" , &cfg->big_pack_threshold);git_config_get_ulong ("pack.deltacachesize" , &cfg->max_delta_cache_size); +git_config_get_ulong ("core.deltabasecachelimit" , &cfg->delta_base_cache_limit);if (!git_config_get_string ("gc.repackfilter" , &owned)) {free (cfg->repack_filter);
@@ -416,7 +419,7 @@ static uint64_t estimate_repack_memory(struct gc_config *cfg,* read_sha1_file() (either at delta calculation phase, or * writing phase) also fills up the delta base cache */ - heap += delta_base_cache_limit; + heap += cfg->delta_base_cache_limit;/* and of course pack-objects has its own delta cache */ heap += cfg->max_delta_cache_size;
builtin/index-pack.c +7 -3
@@ -1238,7 +1238,7 @@ static void parse_pack_objects(unsigned char *hash)* recursively checking if the resulting object is used as a base * for some more deltas. */ -static void resolve_deltas (void ) +static void resolve_deltas (struct pack_idx_option *opts) {int i;
@@ -1254,7 +1254,7 @@ static void resolve_deltas(void) nr_ref_deltas + nr_ofs_deltas); nr_dispatched =0 ; - base_cache_limit = delta_base_cache_limit * nr_threads; + base_cache_limit = opts->delta_base_cache_limit * nr_threads;if (nr_threads >1 ||getenv ("GIT_FORCE_THREADS" )) {init_thread ();work_lock ();
@@ -1604,6 +1604,10 @@ static int git_index_pack_config(const char *k, const char *v,else opts->flags &= ~WRITE_REV ; } +if (!strcmp (k,"core.deltabasecachelimit" )) { + opts->delta_base_cache_limit =git_config_ulong (k, v, ctx->kvi); +return 0 ; + }return git_default_config (k, v, ctx, cb); }
@@ -1930,7 +1934,7 @@ int cmd_index_pack(int argc,parse_pack_objects (pack_hash);if (report_end_of_input)write_in_full (2 ,"\0" ,1 ); -resolve_deltas (); +resolve_deltas (&opts);conclude_pack (fix_thin_pack, curr_pack, pack_hash);free (ofs_deltas);free (ref_deltas);
config.c +0 -5
@@ -1515,11 +1515,6 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.deltabasecachelimit" )) { - delta_base_cache_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -1
@@ -51,7 +51,6 @@ enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ;size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ;size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ; -size_t delta_base_cache_limit =96 *1024 *1024 ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
environment.h +0 -1
@@ -165,7 +165,6 @@ extern int zlib_compression_level;extern int pack_compression_level;extern size_t packed_git_window_size;extern size_t packed_git_limit; -extern size_t delta_base_cache_limit;extern unsigned long big_file_threshold;extern unsigned long pack_size_limit_cfg;extern int max_allowed_tree_depth;
pack-objects.h +2 -1
@@ -7,7 +7,8 @@struct repository ; -#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_BASE_CACHE_LIMIT (96 * 1024 * 1024)#define OE_DFS_STATE_BITS 2#define OE_DEPTH_BITS 12
pack-write.c +1 -0
@@ -21,6 +21,7 @@ void reset_pack_idx_option(struct pack_idx_option *opts)memset (opts,0 ,sizeof (*opts)); opts->version =2 ; opts->off32_limit =0x7fffffff ; + opts->delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; }static int sha1_compare (const void *_a,const void *_b)
pack.h +1 -0
@@ -58,6 +58,7 @@ struct pack_idx_option {*/ int anomaly_alloc, anomaly_nr;uint32_t *anomaly; +unsigned long delta_base_cache_limit; };void reset_pack_idx_option (struct pack_idx_option *);
packfile.c +11 -2
@@ -24,6 +24,8 @@#include "commit-graph.h" #include "pack-revindex.h" #include "promisor-remote.h" +#include "config.h" +#include "pack-objects.h" char *odb_pack_name (struct repository *r,struct strbuf *buf,const unsigned char *hash,const char *ext)
@@ -1496,7 +1498,9 @@ void clear_delta_base_cache(void) }static void add_delta_base_cache (struct packed_git *p,off_t base_offset, -void *base,unsigned long base_size,enum object_type type) +void *base,unsigned long base_size, +unsigned long delta_base_cache_limit, +enum object_type type) {struct delta_base_cache_entry *ent;struct list_head *lru, *tmp;
@@ -1697,6 +1701,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,struct unpack_entry_stack_ent *delta_stack = small_delta_stack;int delta_stack_nr =0 , delta_stack_alloc =UNPACK_ENTRY_STACK_PREALLOC ;int base_from_cache =0 ; +unsigned long delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; + +repo_config_get_ulong (r,"core.deltabasecachelimit" , &delta_base_cache_limit);write_pack_access_log (p, obj_offset);
@@ -1878,7 +1885,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,* before we are done using it. */ if (!external_base) -add_delta_base_cache (p, base_obj_offset, base, base_size, type); +add_delta_base_cache (p, base_obj_offset, base, + base_size, delta_base_cache_limit, + type);free (delta_data);free (external_base);
Signature · 1 line
2.47.0
PATCH v5 8/9 config: make `packed_git_(limit|window_size)` non-global variables
In reply to Karthik NayakThe variables packed_git_window_size and packed_git_limit are global config variables used in the packfile.c file. Since it is only used in this file, let's change it from being a global config variable to a local variable for the subsystem.
We do this by introducing a new local packfile_config struct in packfile.c and also adding the required function to parse the said config. We then use this within packfile.c to obtain the variables.
With this, we rid packfile.c from all global variable usage and this means we can also remove the USE_THE_REPOSITORY_VARIABLE guard from the file.
--- builtin/fast-import.c | 4 ++-- config.c | 17 ----------------- environment.c | 2 -- packfile.c | 19 +++++++++++-------- packfile.h | 2 +- repo-settings.c | 14 ++++++++++++++ repo-settings.h | 5 +++++ 7 files changed, 33 insertions(+), 30 deletions(-)
builtin/fast-import.c +2 -2
@@ -3539,7 +3539,7 @@ static void parse_argv(void)int cmd_fast_import (int argc,const char **argv,const char *prefix, - struct repository *repo UNUSED) +struct repository *repo) {unsigned int i;
@@ -3660,7 +3660,7 @@ int cmd_fast_import(int argc,fprintf (stderr," pools: %10lu KiB\n" , (unsigned long )((tree_entry_allocd + fi_mem_pool.pool_alloc) /1024 ));fprintf (stderr," objects: %10" PRIuMAX" KiB\n" , (alloc_count*sizeof (struct object_entry ))/1024 );fprintf (stderr,"---------------------------------------------------------------------\n" ); -pack_report (); +pack_report (repo);fprintf (stderr,"---------------------------------------------------------------------\n" );fprintf (stderr,"\n" ); }
config.c +0 -17
@@ -1493,28 +1493,11 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.packedgitwindowsize" )) { -int pgsz_x2 =getpagesize () *2 ; - packed_git_window_size =git_config_ulong (var, value, ctx->kvi); - -/* This value must be multiple of (pagesize * 2) */ - packed_git_window_size /= pgsz_x2; -if (packed_git_window_size <1 ) - packed_git_window_size =1 ; - packed_git_window_size *= pgsz_x2; -return 0 ; - } -if (!strcmp (var,"core.bigfilethreshold" )) { big_file_threshold =git_config_ulong (var, value, ctx->kvi);return 0 ; } -if (!strcmp (var,"core.packedgitlimit" )) { - packed_git_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -2
@@ -49,8 +49,6 @@ int fsync_object_files = -1;int use_fsync =-1 ;enum fsync_method fsync_method =FSYNC_METHOD_DEFAULT ;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ; -size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ; -size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
packfile.c +11 -8
@@ -1,4 +1,3 @@ -#define USE_THE_REPOSITORY_VARIABLE #include "git-compat-util.h" #include "environment.h"
@@ -48,15 +47,15 @@ static size_t pack_mapped;#define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; } -void pack_report (void ) +void pack_report (struct repository *repo) {fprintf (stderr,"pack_report: getpagesize() = %10" SZ_FMT "\n" "pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" "pack_report: core.packedGitLimit = %10" SZ_FMT "\n" ,sz_fmt (getpagesize ()), -sz_fmt (packed_git_window_size), -sz_fmt (packed_git_limit)); +sz_fmt (repo->settings.packed_git_window_size), +sz_fmt (repo->settings.packed_git_limit));fprintf (stderr,"pack_report: pack_used_ctr = %10u\n" "pack_report: pack_mmap_calls = %10u\n"
@@ -652,8 +651,11 @@ unsigned char *use_pack(struct packed_git *p,break ; }if (!win) { -size_t window_align = packed_git_window_size /2 ; +size_t window_align;off_t len; +struct repo_settings *settings = &p->repo->settings; + + window_align = settings->packed_git_window_size /2 ;if (p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name);
@@ -661,11 +663,12 @@ unsigned char *use_pack(struct packed_git *p,CALLOC_ARRAY (win,1 ); win->offset = (offset / window_align) * window_align; len = p->pack_size - win->offset; -if (len > packed_git_window_size) - len = packed_git_window_size; +if (len > settings->packed_git_window_size) + len = settings->packed_git_window_size; win->len = (size_t )len; pack_mapped += win->len; -while (packed_git_limit < pack_mapped + +while (settings->packed_git_limit < pack_mapped &&unuse_one_window (p)) ;/* nothing */ win->base = xmmap_gently(NULL, win->len,
packfile.h +1 -1
@@ -89,7 +89,7 @@ unsigned long repo_approximate_object_count(struct repository *r);struct packed_git *find_oid_pack (const struct object_id *oid,struct packed_git *packs); -void pack_report (void ); +void pack_report (struct repository *repo);/* * mmap the index file for the specified packfile (if it is not
repo-settings.c +14 -0
@@ -26,6 +26,7 @@ void prepare_repo_settings(struct repository *r)const char *strval;int manyfiles;int read_changed_paths; +unsigned long longval;if (!r->gitdir)BUG ("Cannot add settings for uninitialized repository" );
@@ -123,6 +124,19 @@ void prepare_repo_settings(struct repository *r)* removed. */ r->settings.command_requires_full_index =1 ; + +if (!repo_config_get_ulong (r,"core.packedgitwindowsize" , &longval)) { +int pgsz_x2 =getpagesize () *2 ; + +/* This value must be multiple of (pagesize * 2) */ + longval /= pgsz_x2; +if (longval <1 ) + longval =1 ; + r->settings.packed_git_window_size = longval * pgsz_x2; + } + +if (!repo_config_get_ulong (r,"core.packedgitlimit" , &longval)) + r->settings.packed_git_limit = longval; }enum log_refs_config repo_settings_get_log_all_ref_updates (struct repository *repo)
repo-settings.h +5 -0
@@ -57,12 +57,17 @@ struct repo_settings {int core_multi_pack_index;int warn_ambiguous_refs;/* lazily loaded via accessor */ + +size_t packed_git_window_size; +size_t packed_git_limit; };#define REPO_SETTINGS_INIT { \ .index_version = -1, \ .core_untracked_cache = UNTRACKED_CACHE_KEEP, \ .fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \ .warn_ambiguous_refs = -1, \ + .packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE, \ + .packed_git_limit = DEFAULT_PACKED_GIT_LIMIT, \ }void prepare_repo_settings (struct repository *r);
Signature · 1 line
2.47.0
Re: [PATCH v5 8/9] config: make `packed_git_(limit|window_size)` non-global variables
Quoted 18 lines · Karthik Nayak
packfile.c +4
-2@@ -652,8 +651,11 @@ unsigned char *use_pack(struct packed_git *p,break ; }if (!win) { -size_t window_align = packed_git_window_size /2 ; +size_t window_align;off_t len; +struct repo_settings *settings = &p->repo->settings; + + window_align = settings->packed_git_window_size /2 ;if (p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name);@@ -661,11 +663,12 @@ unsigned char *use_pack(struct packed_git *p,CALLOC_ARRAY (win,1 ); win->offset = (offset / window_align) * window_align; len = p->pack_size - win->offset; -if (len > packed_git_window_size)
packfile.c unsigned char *use_pack(struct packed_git *p, · line 673
Quoted 10 lines
packfile.c +4
-2- len = packed_git_window_size; +if (len > settings->packed_git_window_size) + len = settings->packed_git_window_size; win->len = (size_t )len; pack_mapped += win->len; -while (packed_git_limit < pack_mapped + +while (settings->packed_git_limit < pack_mapped &&unuse_one_window (p)) ;/* nothing */
Much nicer than the earlier version of the patch.
Do we need to call prepare_repo_settings() here? It looks like the intent is that it would be lazy-loaded, and I don't think there's any guarantee that somebody else would have done so.
Quoted 7 lines
repo-settings.c +3
-0@@ -123,6 +124,19 @@ void prepare_repo_settings(struct repository *r)* removed. */ r->settings.command_requires_full_index =1 ; + +if (!repo_config_get_ulong (r,"core.packedgitwindowsize" , &longval)) { +int pgsz_x2 =getpagesize () *2 ;
repo-settings.c void prepare_repo_settings(struct repository *r) · line 139
Quoted 10 lines
repo-settings.c +10
-0+ +/* This value must be multiple of (pagesize * 2) */ + longval /= pgsz_x2; +if (longval <1 ) + longval =1 ; + r->settings.packed_git_window_size = longval * pgsz_x2; + } + +if (!repo_config_get_ulong (r,"core.packedgitlimit" , &longval)) + r->settings.packed_git_limit = longval;
And this looks like a faithful conversion of the existing parsing. Since we're switching from git_config_ulong() to repo_config_get_ulong(), we could take the opportunity to swap out for the size_t parser, but:
1. I'm just as happy for that to happen separately, and leave this as a patch which should not have any behavior change.
2. It looks like we do not yet have a size_t variant for the configset accessors. :)
-Peff
Re: [PATCH v5 8/9] config: make `packed_git_(limit|window_size)` non-global variables
Quoted 33 lines · Jeff King
Quoted 18 lines · Karthik Nayak
packfile.c +4
-2@@ -652,8 +651,11 @@ unsigned char *use_pack(struct packed_git *p,break ; }if (!win) { -size_t window_align = packed_git_window_size /2 ; +size_t window_align;off_t len; +struct repo_settings *settings = &p->repo->settings; + + window_align = settings->packed_git_window_size /2 ;if (p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name);@@ -661,11 +663,12 @@ unsigned char *use_pack(struct packed_git *p,CALLOC_ARRAY (win,1 ); win->offset = (offset / window_align) * window_align; len = p->pack_size - win->offset; -if (len > packed_git_window_size)packfile.c unsigned char *use_pack(struct packed_git *p, · line 673
Quoted 10 lines
packfile.c +4
-2- len = packed_git_window_size; +if (len > settings->packed_git_window_size) + len = settings->packed_git_window_size; win->len = (size_t )len; pack_mapped += win->len; -while (packed_git_limit < pack_mapped + +while (settings->packed_git_limit < pack_mapped &&unuse_one_window (p)) ;/* nothing */ Much nicer than the earlier version of the patch.
Quoted 4 lines
Do we need to call
prepare_repo_settings()here? It looks like the intent is that it would be lazy-loaded, and I don't think there's any guarantee that somebody else would have done so.
I think it would be safer to do that, than to rely on the tests like I did. I'll change that.
Quoted 22 lines
Quoted 7 lines
repo-settings.c +3
-0@@ -123,6 +124,19 @@ void prepare_repo_settings(struct repository *r)* removed. */ r->settings.command_requires_full_index =1 ; + +if (!repo_config_get_ulong (r,"core.packedgitwindowsize" , &longval)) { +int pgsz_x2 =getpagesize () *2 ;repo-settings.c void prepare_repo_settings(struct repository *r) · line 139
Quoted 10 lines
repo-settings.c +10
-0+ +/* This value must be multiple of (pagesize * 2) */ + longval /= pgsz_x2; +if (longval <1 ) + longval =1 ; + r->settings.packed_git_window_size = longval * pgsz_x2; + } + +if (!repo_config_get_ulong (r,"core.packedgitlimit" , &longval)) + r->settings.packed_git_limit = longval;And this looks like a faithful conversion of the existing parsing. Since we're switching from
git_config_ulong()torepo_config_get_ulong(), we could take the opportunity to swap out for thesize_tparser, but:
Quoted 7 lines
1. I'm just as happy for that to happen separately, and leave this as a patch which should not have any behavior change.
2. It looks like we do not yet have a
size_tvariant for the configset accessors. :)-Peff
Yes, indeed. I'll leave it out of this. I'll follow up if I can! Thanks
Karthik
PATCH v5 9/9 midx: add repository to `multi_pack_index` struct
In reply to Karthik NayakThe multi_pack_index struct represents the MIDX for a repository. Here, we add a pointer to the repository in this struct, allowing direct use of the repository variable without relying on the global the_repository struct.
With this addition, we can determine the repository associated with a bitmap_index struct. A bitmap_index points to either a packed_git or a multi_pack_index, both of which have direct repository references. To support this, we introduce a static helper function, bitmap_repo, in pack-bitmap.c, which retrieves a repository given a bitmap_index.
With this, we clear up all usages of the_repository within pack-bitmap.c and also remove the USE_THE_REPOSITORY_VARIABLE definition. Bringing us another step closer to remove all global variable usage.
Although this change also opens up the potential to clean up midx.c, doing so would require additional refactoring to pass the repository struct to functions where the MIDX struct is created: a task better suited for future patches.
--- midx.c | 1 + midx.h | 3 ++ pack-bitmap.c | 90 +++++++++++++++++++++++++++++++-------------------- 3 files changed, 59 insertions(+), 35 deletions(-)
midx.c +1 -0
@@ -131,6 +131,7 @@ static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir m->data = midx_map; m->data_len = midx_size; m->local = local; + m->repo = the_repository; m->signature =get_be32 (m->data);if (m->signature !=MIDX_SIGNATURE )
midx.h +3 -0
@@ -71,6 +71,9 @@ struct multi_pack_index {const char **pack_names;struct packed_git **packs; + +struct repository *repo; +char object_dir[FLEX_ARRAY ]; };
pack-bitmap.c +55 -35
251 lines not on this page · raw message
Signature · 1 line
2.47.0
Quoted 8 lines · Karthik Nayak
Range-diff against v4:
1: b3d518e998 = 1: 6c00e25c86 packfile: add repository to struct `packed_git` 2: bb9d9aa744 = 2: 70fc8a79af packfile: use `repository` from `packed_git` directly 3: d5df50fa36 = 3: 167a1f3a11 packfile: pass `repository` to static function in the file 4: 0107801c3b = 4: b7cfe78217 packfile: pass down repository to `odb_pack_name` 5: 2d7608a367 = 5: 5566f5554c packfile: pass down repository to `has_object[_kept]_pack` 6: 2c84026d02 = 6: 1b26e45a9b packfile: pass down repository to `for_each_packed_object` 7: 84b89c8a0e ! 7: 7654bf5e7e config: make `delta_base_cache_limit` a non-global variable
Quoted 8 lines
@@ Commit message
this value from the repository config, since the value is only used once in the entire subsystem.+ The type of the value is changed from
size_tto anunsigned long
+ since the default value is small enough to fit inside the latter on all
+ platforms.
+
I think this change is not ideal, for the same reason that the other type changes were: you can conceivably have a 4GB or larger cache here. On Windows using "unsigned long" would prevent that. (On most other systems it is OK either way since "unsigned long" and "size_t" are generally the same size).
I do think the config parsing should change to use size_t here (like I mentioned elsewhere in the thread), which would fix it on Windows. That's outside the scope of your patch, but in the meantime we should not be making things worse by moving the variable itself to the inferior type.
-Peff
Quoted 25 lines · Jeff King
Quoted 8 lines · Karthik Nayak
Range-diff against v4:
1: b3d518e998 = 1: 6c00e25c86 packfile: add repository to struct `packed_git` 2: bb9d9aa744 = 2: 70fc8a79af packfile: use `repository` from `packed_git` directly 3: d5df50fa36 = 3: 167a1f3a11 packfile: pass `repository` to static function in the file 4: 0107801c3b = 4: b7cfe78217 packfile: pass down repository to `odb_pack_name` 5: 2d7608a367 = 5: 5566f5554c packfile: pass down repository to `has_object[_kept]_pack` 6: 2c84026d02 = 6: 1b26e45a9b packfile: pass down repository to `for_each_packed_object` 7: 84b89c8a0e ! 7: 7654bf5e7e config: make `delta_base_cache_limit` a non-global variableQuoted 8 lines
@@ Commit message
this value from the repository config, since the value is only used once in the entire subsystem.+ The type of the value is changed from
size_tto anunsigned long
+ since the default value is small enough to fit inside the latter on all
+ platforms.
+I think this change is not ideal, for the same reason that the other type changes were: you can conceivably have a 4GB or larger cache here. On Windows using "unsigned long" would prevent that. (On most other systems it is OK either way since "unsigned long" and "
size_t" are generally the same size).
Quoted 6 lines
I do think the config parsing should change to use
size_there (like I mentioned elsewhere in the thread), which would fix it on Windows. That's outside the scope of your patch, but in the meantime we should not be making things worse by moving the variable itself to the inferior type.
There is a subtle difference though. Both the configs (this and next commit) although are initialized with 'size_t' they are bounded by 'unsigned long's range, since they do to the functions used to obtain the information. The only difference being the defaults, which could've been greater than the range of 'unsigned long'.
While having said that, keeping it 'size_t' makes it easier to know which configs need to use the yet_to_be introduced 'size_t' variants of the config parsers. So let me change this too. Thanks.
Karthik
v6 cover packfile: avoid using the 'the_repository' global variable
In reply to Karthik NayakThe packfile.c file uses the global variable 'the_repository' extensively throughout the code. Let's remove all usecases of this, by modifying the required functions to accept a 'struct repository' instead. This is to clean up usage of global state.
The first 3 patches are mostly internal to packfile.c, we add the repository field to the packed_git struct and this is used to clear up some useages of the global variables.
The next 3 patches are more disruptive, they modify the function definition of odb_pack_name, has_object[_kept]_pack and for_each_packed_object to receive a repository, helping remove other usages of 'the_repository' variable.
Finally, the last two patches deal with global config values. These values are localized.
For v5/6 onwards, I've rebased the series off the new master: 8f8d6eee53 (The seventh batch, 2024-11-01), as a dependency for this series 'jk/dumb-http-finalize' was merged to master. I've found no conflicts while merging with seen & next. But since this series does touch multiple files, there could be future conflicts.
Changes in v6:
- Lazy load repository settings in packfile.c. This ensures that the settings are available for sure and we do not rely on callees setting it.
- Use size_t for delta_base_cache_limit.
Changes in v5:
- Move packed_git* settings to repo_settings to ensure we don't keep reparsing the settings in use_pack.
Changes in v4:
- Renamed the repository field within packed_git and multi_pack_index from r to repo, while keeping function parameters to be r.
- Fixed bad braces.
Changes in v3:
- Improved commit messages. In the first commit to talk about how packed_git struct could also be part of the alternates of a repository. In the 7th commit to talk about the motive behind removing the global variable.
- Changed 'packed_git->repo' to 'packed_git->r' to keep it consistent with the rest of the code base.
- Replaced 'the_repository' with locally available access to the repository struct in multiple regions.
- Removed unecessary inclusion of the 'repository.h' header file by forward declaring the 'repository' struct.
- Replace memcpy with hashcpy.
- Change the logic in the 7th patch to use if else statements.
- Added an extra commit to cleanup pack-bitmap.c.
Karthik Nayak (9):
packfile: add repository to struct packed_git
packfile: use repository from packed_git directly
packfile: pass repository to static function in the file
packfile: pass down repository to odb_pack_name
packfile: pass down repository to has_object[_kept]_pack
packfile: pass down repository to for_each_packed_object
config: make delta_base_cache_limit a non-global variable
config: make packed_git_(limit|window_size) non-global variables
midx: add repository to multi_pack_index struct
builtin/cat-file.c | 7 +- builtin/count-objects.c | 2 +- builtin/fast-import.c | 15 ++-- builtin/fsck.c | 20 +++--- builtin/gc.c | 8 ++- builtin/index-pack.c | 20 ++++-- builtin/pack-objects.c | 11 +-- builtin/pack-redundant.c | 2 +- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 4 +- config.c | 22 ------ connected.c | 3 +- diff.c | 3 +- environment.c | 3 - environment.h | 1 - fsck.c | 2 +- http.c | 4 +- list-objects.c | 7 +- midx-write.c | 2 +- midx.c | 3 +- midx.h | 3 + object-store-ll.h | 9 ++- pack-bitmap.c | 90 +++++++++++++++--------- pack-objects.h | 3 +- pack-write.c | 1 + pack.h | 1 + packfile.c | 148 +++++++++++++++++++++++---------------- packfile.h | 18 +++-- promisor-remote.c | 2 +- prune-packed.c | 2 +- reachable.c | 4 +- repo-settings.c | 14 ++++ repo-settings.h | 5 ++ revision.c | 13 ++-- tag.c | 2 +- 36 files changed, 268 insertions(+), 190 deletions(-)
Range-diff against v5:
-: ---------- > 1: 6c00e25c86 packfile: add repository to struct `packed_git` -: ---------- > 2: 70fc8a79af packfile: use `repository` from `packed_git` directly -: ---------- > 3: 167a1f3a11 packfile: pass `repository` to static function in the file -: ---------- > 4: b7cfe78217 packfile: pass down repository to `odb_pack_name` -: ---------- > 5: 5566f5554c packfile: pass down repository to `has_object[_kept]_pack` -: ---------- > 6: 1b26e45a9b packfile: pass down repository to `for_each_packed_object` 1: 7654bf5e7e ! 7: 89313cfed4 config: make `delta_base_cache_limit` a non-global variable
@@ Commit message
this value from the repository config, since the value is only used once in the entire subsystem.
- The type of the value is changed from size_t to an unsigned long
- since the default value is small enough to fit inside the latter on all
- platforms.
-
These changes are made to remove the usage of delta_base_cache_limit as a global variable in packfile.c. This brings us one step closer to removing the USE_THE_REPOSITORY_VARIABLE definition in packfile.c
@@ builtin/gc.c: struct gc_config {
char *repack_filter_to;
unsigned long big_pack_threshold;
unsigned long max_delta_cache_size;
-+ unsigned long delta_base_cache_limit;
++ size_t delta_base_cache_limit;
};
#define GC_CONFIG_INIT { \
@@ builtin/gc.c: struct gc_config {
static void gc_config_release(struct gc_config *cfg)
@@ builtin/gc.c: static void gc_config(struct gc_config *cfg)
+ {
+ const char *value;
+ char *owned = NULL;
++ unsigned long longval;
+ if (!git_config_get_value("gc.packrefs", &value)) {
+ if (value && !strcmp(value, "notbare"))
+@@ builtin/gc.c: static void gc_config(struct gc_config *cfg)
git_config_get_ulong("gc.bigpackthreshold", &cfg->big_pack_threshold); git_config_get_ulong("pack.deltacachesize", &cfg->max_delta_cache_size);
-+ git_config_get_ulong("core.deltabasecachelimit", &cfg->delta_base_cache_limit);
++ if(!git_config_get_ulong("core.deltabasecachelimit", &longval))
++ cfg->delta_base_cache_limit = longval;
++
if (!git_config_get_string("gc.repackfilter", &owned)) {
free(cfg->repack_filter);
+ cfg->repack_filter = owned;
@@ builtin/gc.c: static uint64_t estimate_repack_memory(struct gc_config *cfg,
* read_sha1_file() (either at delta calculation phase, or
* writing phase) also fills up the delta base cache
2: 2730aacd8e ! 8: 3a8e3b88df config: make packed_git_(limit|window_size) non-global variables
@@ packfile.c: unsigned char *use_pack(struct packed_git *p,
- size_t window_align = packed_git_window_size / 2;
+ size_t window_align;
off_t len;
-+ struct repo_settings *settings = &p->repo->settings;
++ struct repo_settings *settings;
++
++ /* lazy load the settings incase it hasn't been setup */
++ prepare_repo_settings(p->repo);
++ settings = &p->repo->settings;
+
+ window_align = settings->packed_git_window_size / 2; 3: 8e33d40077 = 9: 2f9a146978 midx: add repository to multi_pack_index struct
Signature · 1 line
2.47.0
PATCH v6 1/9 packfile: add repository to struct `packed_git`
The struct packed_git holds information regarding a packed object file. Let's add the repository variable to this object, to represent the repository that this packfile belongs to. This helps remove dependency on the global the_repository object in packfile.c by simply using repository information now readily available in the struct.
We do need to consider that a pack file could be part of the alternates of a repository, but considering that we only have one repository struct and also that we currently anyways use 'the_repository'. We should be OK with this change.
We also modify alloc_packed_git to ensure that the repository is added to newly created packed_git structs. This requires modifying the function and all its callee to pass the repository object down the levels.
--- builtin/fast-import.c | 3 ++- builtin/index-pack.c | 6 ++++-- commit-graph.c | 2 +- connected.c | 3 ++- http.c | 2 +- midx-write.c | 2 +- midx.c | 2 +- object-store-ll.h | 5 +++++ packfile.c | 15 +++++++++------ packfile.h | 6 ++++-- 10 files changed, 30 insertions(+), 16 deletions(-)
builtin/fast-import.c +2 -1
@@ -765,6 +765,7 @@ static void start_packfile(void) p->pack_fd = pack_fd; p->do_not_close =1 ; + p->repo = the_repository; pack_file =hashfd (pack_fd, p->pack_name); pack_data = p;
@@ -888,7 +889,7 @@ static void end_packfile(void) idx_name =keep_pack (create_index ());/* Register the packfile with core git's machinery. */ - new_p =add_packed_git (idx_name,strlen (idx_name),1 ); + new_p =add_packed_git (pack_data->repo, idx_name,strlen (idx_name),1 );if (!new_p)die ("core git rejected index %s" , idx_name); all_packs[pack_id] = new_p;
builtin/index-pack.c +4 -2
@@ -1552,7 +1552,8 @@ static void final(const char *final_pack_name, const char *curr_pack_name,if (do_fsck_object) {struct packed_git *p; - p =add_packed_git (final_index_name,strlen (final_index_name),0 ); + p =add_packed_git (the_repository, final_index_name, +strlen (final_index_name),0 );if (p)install_packed_git (the_repository, p); }
@@ -1650,7 +1651,8 @@ static void read_v2_anomalous_offsets(struct packed_git *p,static void read_idx_option (struct pack_idx_option *opts,const char *pack_name) { -struct packed_git *p =add_packed_git (pack_name,strlen (pack_name),1 ); +struct packed_git *p =add_packed_git (the_repository, pack_name, +strlen (pack_name),1 );if (!p)die (_ ("Cannot open existing pack file '%s'" ), pack_name);
commit-graph.c +1 -1
@@ -1914,7 +1914,7 @@ static int fill_oids_from_packs(struct write_commit_graph_context *ctx,struct packed_git *p;strbuf_setlen (&packname, dirlen);strbuf_addstr (&packname, pack_indexes->items[i].string); - p =add_packed_git (packname.buf, packname.len,1 ); + p =add_packed_git (ctx->r, packname.buf, packname.len,1 );if (!p) { ret =error (_ ("error adding pack %s" ), packname.buf); goto cleanup;
connected.c +2 -1
@@ -54,7 +54,8 @@ int check_connected(oid_iterate_fn fn, void *cb_data,strbuf_add (&idx_file, transport->pack_lockfiles.items[0 ].string, base_len);strbuf_addstr (&idx_file,".idx" ); - new_pack =add_packed_git (idx_file.buf, idx_file.len,1 ); + new_pack =add_packed_git (the_repository, idx_file.buf, + idx_file.len,1 );strbuf_release (&idx_file); }
http.c +1 -1
@@ -2439,7 +2439,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,if (!tmp_idx)return -1 ; - new_pack =parse_pack_index (sha1, tmp_idx); + new_pack =parse_pack_index (the_repository, sha1, tmp_idx);if (!new_pack) {unlink (tmp_idx);free (tmp_idx);
midx-write.c +1 -1
@@ -154,7 +154,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,return ;ALLOC_GROW (ctx->info, ctx->nr +1 , ctx->alloc); - p =add_packed_git (full_path, full_path_len,0 ); + p =add_packed_git (the_repository, full_path, full_path_len,0 );if (!p) {warning (_ ("failed to add packfile '%s'" ), full_path);
midx.c +1 -1
@@ -464,7 +464,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,strhash (key.buf), key.buf, struct packed_git, packmap_ent);if (!p) { - p =add_packed_git (pack_name.buf, pack_name.len, m->local); + p =add_packed_git (r, pack_name.buf, pack_name.len, m->local);if (p) {install_packed_git (r, p);list_add_tail (&p->mru, &r->objects->packed_git_mru);
object-store-ll.h +5 -0
@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;struct object_directory {struct object_directory *next;
@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo dentoes the repository this packed file belongs to */ +struct repository *repo; +/* something like ".git/objects/pack/xxxxx.pack" */ char pack_name[FLEX_ARRAY ];/* more */ };
packfile.c +9 -6
@@ -217,11 +217,12 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)return ntohl (level1_ofs[value]); } -static struct packed_git *alloc_packed_git (int extra) +static struct packed_git *alloc_packed_git (struct repository *r,int extra) {struct packed_git *p =xmalloc (st_add (sizeof (*p), extra));memset (p,0 ,sizeof (*p)); p->pack_fd =-1 ; + p->repo = r;return p; }
@@ -233,11 +234,12 @@ static char *pack_path_from_idx(const char *idx_path)return xstrfmt ("%.*s.pack" , (int )len, idx_path); } -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path) {char *path =pack_path_from_idx (idx_path);size_t alloc =st_add (strlen (path),1 ); -struct packed_git *p =alloc_packed_git (alloc); +struct packed_git *p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path);
@@ -703,7 +705,8 @@ void unuse_pack(struct pack_window **w_cursor) } } -struct packed_git *add_packed_git (const char *path,size_t path_len,int local) +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local) {struct stat st;size_t alloc;
@@ -721,7 +724,7 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)* the use xsnprintf double-checks that) */ alloc =st_add3 (path_len,strlen (".promisor" ),1 ); - p =alloc_packed_git (alloc); + p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, path_len);xsnprintf (p->pack_name + path_len, alloc - path_len,".keep" );
@@ -877,7 +880,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,/* Don't reopen a pack we already have. */ if (!hashmap_get (&data->r->objects->pack_map, &hent, pack_name)) { - p =add_packed_git (full_name, full_name_len, data->local); + p =add_packed_git (data->r, full_name, full_name_len, data->local);if (p)install_packed_git (data->r, p); }
packfile.h +4 -2
@@ -46,7 +46,8 @@ const char *pack_basename(struct packed_git *p);* and does not add the resulting packed_git struct to the internal list of * packs. You probably want add_packed_git() instead. */ -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path);typedef void each_file_in_pack_dir_fn (const char *full_path,size_t full_path_len,const char *file_name,void *data);
@@ -113,7 +114,8 @@ void close_pack(struct packed_git *);void close_object_store (struct raw_object_store *o);void unuse_pack (struct pack_window **);void clear_delta_base_cache (void ); -struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local);/* * Unlink the .pack and associated extension files.
Signature · 1 line
2.47.0
PATCH v6 2/9 packfile: use `repository` from `packed_git` directly
In reply to Karthik NayakIn the previous commit, we introduced the repository structure inside packed_git. This provides an alternative route instead of using the global the_repository variable. Let's modify packfile.c now to use this field wherever possible instead of relying on the global state. There are still a few instances of the_repository usage in the file, where there is no struct packed_git locally available, which will be fixed in the following commits.
--- packfile.c | 50 +++++++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 23 deletions(-)
packfile.c +27 -23
@@ -79,7 +79,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)size_t idx_size;int fd =git_open (path), ret;struct stat st; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (fd <0 )return -1 ;
@@ -243,7 +243,7 @@ struct packed_git *parse_pack_index(struct repository *r, unsigned char *sha1,memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path); -hashcpy (p->hash, sha1, the_repository->hash_algo); +hashcpy (p->hash, sha1, p->repo->hash_algo);if (check_packed_git_idx (idx_path, p)) {free (p);return NULL ;
@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = the_repository->objects->packed_git; p; p = p->next) +for (p = current->repo->objects->packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -540,7 +540,7 @@ static int open_packed_git_1(struct packed_git *p)unsigned char hash[GIT_MAX_RAWSZ ];unsigned char *idx_hash;ssize_t read_result; -const unsigned hashsz = the_hash_algo->rawsz; +const unsigned hashsz = p->repo->hash_algo->rawsz;if (open_pack_index (p))return error ("packfile %s index unavailable" , p->pack_name);
@@ -597,7 +597,7 @@ static int open_packed_git_1(struct packed_git *p)if (read_result != hashsz)return error ("packfile %s signature is unavailable" , p->pack_name); idx_hash = ((unsigned char *)p->index_data) + p->index_size - hashsz *2 ; -if (!hasheq (hash, idx_hash, the_repository->hash_algo)) +if (!hasheq (hash, idx_hash, p->repo->hash_algo))return error ("packfile %s does not match index" , p->pack_name);return 0 ; }
@@ -637,7 +637,7 @@ unsigned char *use_pack(struct packed_git *p,*/ if (!p->pack_size && p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name); -if (offset > (p->pack_size - the_hash_algo->rawsz)) +if (offset > (p->pack_size - p->repo->hash_algo->rawsz))die ("offset beyond end of packfile (truncated pack?)" );if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" ));
@@ -711,6 +711,7 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,struct stat st;size_t alloc;struct packed_git *p; +struct object_id oid;/* * Make sure a corresponding .pack file exists and that
@@ -751,9 +752,13 @@ struct packed_git *add_packed_git(struct repository *r, const char *path, p->pack_size = st.st_size; p->pack_local = local; p->mtime = st.st_mtime; -if (path_len < the_hash_algo->hexsz || -get_hash_hex (path + path_len - the_hash_algo->hexsz, p->hash)) -hashclr (p->hash, the_repository->hash_algo); +if (path_len < r->hash_algo->hexsz || +get_oid_hex_algop (path + path_len - r->hash_algo->hexsz, &oid, + r->hash_algo)) +hashclr (p->hash, r->hash_algo); +else +hashcpy (p->hash, oid.hash, r->hash_algo); +return p; }
@@ -1243,9 +1248,9 @@ off_t get_delta_base(struct packed_git *p, }else if (type ==OBJ_REF_DELTA ) {/* The base entry _must_ be in the same pack */ struct object_id oid; -oidread (&oid, base_info, the_repository->hash_algo); +oidread (&oid, base_info, p->repo->hash_algo); base_offset =find_pack_entry_one (&oid, p); - *curpos += the_hash_algo->rawsz; + *curpos += p->repo->hash_algo->rawsz; }else die ("I am totally screwed" );return base_offset;
@@ -1266,7 +1271,7 @@ static int get_delta_base_oid(struct packed_git *p, {if (type ==OBJ_REF_DELTA ) {unsigned char *base =use_pack (p, w_curs, curpos,NULL ); -oidread (oid, base, the_repository->hash_algo); +oidread (oid, base, p->repo->hash_algo);return 0 ; }else if (type ==OBJ_OFS_DELTA ) {uint32_t base_pos;
@@ -1608,7 +1613,7 @@ int packed_object_info(struct repository *r, struct packed_git *p, goto out; } }else -oidclr (oi->delta_base_oid, the_repository->hash_algo); +oidclr (oi->delta_base_oid, p->repo->hash_algo); } oi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :
@@ -1897,7 +1902,7 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32 {const unsigned char *index_fanout = p->index_data;const unsigned char *index_lookup; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;int index_lookup_width;if (!index_fanout)
@@ -1922,7 +1927,7 @@ int nth_packed_object_id(struct object_id *oid, uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (!index) {if (open_pack_index (p))return -1 ;
@@ -1933,11 +1938,10 @@ int nth_packed_object_id(struct object_id *oid, index +=4 *256 ;if (p->index_version ==1 ) {oidread (oid, index +st_add (st_mult (hashsz +4 , n),4 ), - the_repository->hash_algo); + p->repo->hash_algo); }else { index +=8 ; -oidread (oid, index +st_mult (hashsz, n), - the_repository->hash_algo); +oidread (oid, index +st_mult (hashsz, n), p->repo->hash_algo); }return 0 ; }
@@ -1959,7 +1963,7 @@ void check_pack_index_ptr(const struct packed_git *p, const void *vptr)off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz; index +=4 *256 ;if (p->index_version ==1 ) {return ntohl (*((uint32_t *)(index +st_mult (hashsz +4 , n))));
@@ -2159,7 +2163,7 @@ int for_each_object_in_pack(struct packed_git *p,int r =0 ;if (flags &FOR_EACH_OBJECT_PACK_ORDER ) { -if (load_pack_revindex (the_repository, p)) +if (load_pack_revindex (p->repo, p))return -1 ; }
@@ -2227,7 +2231,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, }static int add_promisor_object (const struct object_id *oid, - struct packed_git *pack UNUSED, +struct packed_git *pack, uint32_t pos UNUSED,void *set_) {
@@ -2235,12 +2239,12 @@ static int add_promisor_object(const struct object_id *oid,struct object *obj;int we_parsed_object; - obj =lookup_object (the_repository, oid); + obj =lookup_object (pack->repo, oid);if (obj && obj->parsed) { we_parsed_object =0 ; }else { we_parsed_object =1 ; - obj =parse_object (the_repository, oid); + obj =parse_object (pack->repo, oid); }if (!obj)
Signature · 1 line
2.47.0
PATCH v6 3/9 packfile: pass `repository` to static function in the file
In reply to Karthik NayakSome of the static functions in the packfile.c access global variables, which can simply be avoiding by passing the repository struct down to them. Let's do that.
--- packfile.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
packfile.c +8 -7
@@ -460,13 +460,13 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc *accept_windows_inuse = has_windows_inuse; } -static int close_one_pack (void ) +static int close_one_pack (struct repository *r) {struct packed_git *p, *lru_p =NULL ;struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = the_repository->objects->packed_git; p; p = p->next) { +for (p = r->objects->packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -555,7 +555,7 @@ static int open_packed_git_1(struct packed_git *p) pack_max_fds =1 ; } -while (pack_max_fds <= pack_open_fds &&close_one_pack ()) +while (pack_max_fds <= pack_open_fds &&close_one_pack (p->repo)) ;/* nothing */ p->pack_fd =git_open (p->pack_name);
@@ -610,7 +610,8 @@ static int open_packed_git(struct packed_git *p)return -1 ; } -static int in_window (struct pack_window *win,off_t offset) +static int in_window (struct repository *r,struct pack_window *win, +off_t offset) {/* We must promise at least one full hash after the * offset is available from this window, otherwise the offset
@@ -620,7 +621,7 @@ static int in_window(struct pack_window *win, off_t offset)*/ off_t win_off = win->offset;return win_off <= offset - && (offset + the_hash_algo->rawsz) <= (win_off + win->len); + && (offset + r->hash_algo->rawsz) <= (win_off + win->len); } unsigned char *use_pack(struct packed_git *p,
@@ -642,11 +643,11 @@ unsigned char *use_pack(struct packed_git *p,if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" )); -if (!win || !in_window (win, offset)) { +if (!win || !in_window (p->repo, win, offset)) {if (win) win->inuse_cnt--;for (win = p->windows; win; win = win->next) { -if (in_window (win, offset)) +if (in_window (p->repo, win, offset))break ; }if (!win) {
Signature · 1 line
2.47.0
PATCH v6 4/9 packfile: pass down repository to `odb_pack_name`
In reply to Karthik NayakThe function odb_pack_name currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers.
--- builtin/fast-import.c | 8 ++++---- builtin/index-pack.c | 4 ++-- builtin/pack-redundant.c | 2 +- http.c | 2 +- packfile.c | 9 ++++----- packfile.h | 3 ++- 6 files changed, 14 insertions(+), 14 deletions(-)
builtin/fast-import.c +4 -4
@@ -806,7 +806,7 @@ static char *keep_pack(const char *curr_index_name)struct strbuf name =STRBUF_INIT ;int keep_fd; -odb_pack_name (&name, pack_data->hash,"keep" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"keep" ); keep_fd =odb_pack_keep (name.buf);if (keep_fd <0 )die_errno ("cannot create keep file" );
@@ -814,11 +814,11 @@ static char *keep_pack(const char *curr_index_name)if (close (keep_fd))die_errno ("failed to write keep file" ); -odb_pack_name (&name, pack_data->hash,"pack" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"pack" );if (finalize_object_file (pack_data->pack_name, name.buf))die ("cannot store pack file" ); -odb_pack_name (&name, pack_data->hash,"idx" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"idx" );if (finalize_object_file (curr_index_name, name.buf))die ("cannot store index file" );free ((void *)curr_index_name);
@@ -832,7 +832,7 @@ static void unkeep_all_packs(void)for (k =0 ; k < pack_id; k++) {struct packed_git *p = all_packs[k]; -odb_pack_name (&name, p->hash,"keep" ); +odb_pack_name (p->repo, &name, p->hash,"keep" );unlink_or_warn (name.buf); }strbuf_release (&name);
builtin/index-pack.c +2 -2
@@ -1479,7 +1479,7 @@ static void write_special_file(const char *suffix, const char *msg,if (pack_name) filename =derive_filename (pack_name,"pack" , suffix, &name_buf);else - filename =odb_pack_name (&name_buf, hash, suffix); + filename =odb_pack_name (the_repository, &name_buf, hash, suffix); fd =odb_pack_keep (filename);if (fd <0 ) {
@@ -1507,7 +1507,7 @@ static void rename_tmp_packfile(const char **final_name, {if (!*final_name ||strcmp (*final_name, curr_name)) {if (!*final_name) - *final_name =odb_pack_name (name, hash, ext); + *final_name =odb_pack_name (the_repository, name, hash, ext);if (finalize_object_file (curr_name, *final_name))die (_ ("unable to rename temporary '*.%s' file to '%s'" ), ext, *final_name);
builtin/pack-redundant.c +1 -1
@@ -690,7 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix UNUSED, s pl = red =pack_list_difference (local_packs, min);while (pl) {printf ("%s\n%s\n" , -odb_pack_name (&idx_name, pl->pack->hash,"idx" ), +odb_pack_name (pl->pack->repo, &idx_name, pl->pack->hash,"idx" ), pl->pack->pack_name); pl = pl->next; }
http.c +1 -1
@@ -2581,7 +2581,7 @@ struct http_pack_request *new_direct_http_pack_request( preq->url = url; -odb_pack_name (&preq->tmpfile, packed_git_hash,"pack" ); +odb_pack_name (the_repository, &preq->tmpfile, packed_git_hash,"pack" );strbuf_addstr (&preq->tmpfile,".temp" ); preq->packfile =fopen (preq->tmpfile.buf,"a" );if (!preq->packfile) {
packfile.c +4 -5
@@ -25,13 +25,12 @@#include "pack-revindex.h" #include "promisor-remote.h" -char *odb_pack_name (struct strbuf *buf, -const unsigned char *hash, -const char *ext) +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext) {strbuf_reset (buf); -strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (the_repository), -hash_to_hex (hash), ext); +strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (r), +hash_to_hex_algop (hash, r->hash_algo), ext);return buf->buf; }
packfile.h +2 -1
@@ -29,7 +29,8 @@ struct pack_entry {* * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" */ -char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext);/* * Return the basename of the packfile, omitting any containing directory
Signature · 1 line
2.47.0
PATCH v6 5/9 packfile: pass down repository to `has_object[_kept]_pack`
In reply to Karthik NayakThe functions has_object[_kept]_pack currently rely on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from these functions and any related ones.
--- builtin/count-objects.c | 2 +- builtin/fsck.c | 2 +- builtin/pack-objects.c | 4 ++-- diff.c | 3 ++- list-objects.c | 3 ++- pack-bitmap.c | 2 +- packfile.c | 9 +++++---- packfile.h | 5 +++-- prune-packed.c | 2 +- reachable.c | 2 +- revision.c | 4 ++-- 11 files changed, 21 insertions(+), 17 deletions(-)
builtin/count-objects.c +1 -1
@@ -67,7 +67,7 @@ static int count_loose(const struct object_id *oid, const char *path,else { loose_size +=on_disk_bytes (st); loose++; -if (verbose &&has_object_pack (oid)) +if (verbose &&has_object_pack (the_repository, oid)) packed_loose++; }return 0 ;
builtin/fsck.c +1 -1
@@ -272,7 +272,7 @@ static void check_reachable_object(struct object *obj)if (!(obj->flags &HAS_OBJ )) {if (is_promisor_object (&obj->oid))return ; -if (has_object_pack (&obj->oid)) +if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */ printf_ln(_("missing %s %s"), printable_type(&obj->oid, obj->type),
builtin/pack-objects.c +2 -2
@@ -1529,7 +1529,7 @@ static int want_found_object(const struct object_id *oid, int exclude,return 0 ;if (ignore_packed_keep_in_core && p->pack_keep_in_core)return 0 ; -if (has_object_kept_pack (oid, flags)) +if (has_object_kept_pack (p->repo, oid, flags))return 0 ; }
@@ -3627,7 +3627,7 @@ static void show_cruft_commit(struct commit *commit, void *data) static int cruft_include_check_obj(struct object *obj, void *data UNUSED) { -return !has_object_kept_pack (&obj->oid,IN_CORE_KEEP_PACKS ); +return !has_object_kept_pack (to_pack.repo, &obj->oid,IN_CORE_KEEP_PACKS ); }static int cruft_include_check (struct commit *commit,void *data)
diff.c +2 -1
@@ -4041,7 +4041,8 @@ static int reuse_worktree_file(struct index_state *istate,* objects however would tend to be slower as they need * to be individually opened and inflated. */ -if (!FAST_WORKING_DIRECTORY && !want_file &&has_object_pack (oid)) +if (!FAST_WORKING_DIRECTORY && !want_file && +has_object_pack (istate->repo, oid))return 0 ;/*
list-objects.c +2 -1
@@ -41,7 +41,8 @@ static void show_object(struct traversal_context *ctx, {if (!ctx->show_object)return ; -if (ctx->revs->unpacked &&has_object_pack (&object->oid)) +if (ctx->revs->unpacked &&has_object_pack (ctx->revs->repo, + &object->oid))return ; ctx->show_object (object, name, ctx->show_data);
pack-bitmap.c +1 -1
@@ -1889,7 +1889,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,bitmap_unset (result, i);for (i =0 ; i < eindex->count; ++i) { -if (has_object_pack (&eindex->objects[i]->oid)) +if (has_object_pack (the_repository, &eindex->objects[i]->oid))bitmap_unset (result, objects_nr + i); } }
packfile.c +5 -4
@@ -2143,16 +2143,17 @@ int find_kept_pack_entry(struct repository *r,return 0 ; } -int has_object_pack (const struct object_id *oid) +int has_object_pack (struct repository *r,const struct object_id *oid) {struct pack_entry e; -return find_pack_entry (the_repository, oid, &e); +return find_pack_entry (r, oid, &e); } -int has_object_kept_pack (const struct object_id *oid,unsigned flags) +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags) {struct pack_entry e; -return find_kept_pack_entry (the_repository, oid, flags, &e); +return find_kept_pack_entry (r, oid, flags, &e); } int for_each_object_in_pack(struct packed_git *p,
packfile.h +3 -2
@@ -193,8 +193,9 @@ const struct packed_git *has_packed_and_bad(struct repository *, const struct obint find_pack_entry (struct repository *r,const struct object_id *oid,struct pack_entry *e);int find_kept_pack_entry (struct repository *r,const struct object_id *oid,unsigned flags,struct pack_entry *e); -int has_object_pack (const struct object_id *oid); -int has_object_kept_pack (const struct object_id *oid,unsigned flags); +int has_object_pack (struct repository *r,const struct object_id *oid); +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags);/* * Return 1 if an object in a promisor packfile is or refers to the given
prune-packed.c +1 -1
@@ -24,7 +24,7 @@ static int prune_object(const struct object_id *oid, const char *path, {int *opts = data; -if (!has_object_pack (oid)) +if (!has_object_pack (the_repository, oid))return 0 ;if (*opts &PRUNE_PACKED_DRY_RUN )
reachable.c +1 -1
@@ -239,7 +239,7 @@ static int want_recent_object(struct recent_data *data, const struct object_id *oid) {if (data->ignore_in_core_kept_packs && -has_object_kept_pack (oid,IN_CORE_KEEP_PACKS )) +has_object_kept_pack (data->revs->repo, oid,IN_CORE_KEEP_PACKS ))return 0 ;return 1 ; }
revision.c +2 -2
@@ -4103,10 +4103,10 @@ enum commit_action get_commit_action(struct rev_info *revs, struct commit *commi {if (commit->object.flags &SHOWN )return commit_ignore; -if (revs->unpacked &&has_object_pack (&commit->object.oid)) +if (revs->unpacked &&has_object_pack (revs->repo, &commit->object.oid))return commit_ignore;if (revs->no_kept_objects) { -if (has_object_kept_pack (&commit->object.oid, +if (has_object_kept_pack (revs->repo, &commit->object.oid, revs->keep_pack_cache_flags))return commit_ignore; }
Signature · 1 line
2.47.0
PATCH v6 6/9 packfile: pass down repository to `for_each_packed_object`
In reply to Karthik NayakThe function for_each_packed_object currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from this function and closely related function is_promisor_object.
--- builtin/cat-file.c | 7 ++++--- builtin/fsck.c | 18 +++++++++++------- builtin/pack-objects.c | 7 +++++-- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 2 +- fsck.c | 2 +- list-objects.c | 4 ++-- object-store-ll.h | 4 ++-- packfile.c | 14 +++++++------- packfile.h | 2 +- promisor-remote.c | 2 +- reachable.c | 2 +- revision.c | 9 +++++---- tag.c | 2 +- 15 files changed, 44 insertions(+), 35 deletions(-)
builtin/cat-file.c +4 -3
@@ -827,15 +827,16 @@ static int batch_objects(struct batch_options *opt) cb.seen = &seen;for_each_loose_object (batch_unordered_loose, &cb,0 ); -for_each_packed_object (batch_unordered_packed, &cb, -FOR_EACH_OBJECT_PACK_ORDER ); +for_each_packed_object (the_repository, batch_unordered_packed, + &cb,FOR_EACH_OBJECT_PACK_ORDER );oidset_clear (&seen); }else {struct oid_array sa =OID_ARRAY_INIT ;for_each_loose_object (collect_loose_object, &sa,0 ); -for_each_packed_object (collect_packed_object, &sa,0 ); +for_each_packed_object (the_repository, collect_packed_object, + &sa,0 );oid_array_for_each_unique (&sa, batch_object_cb, &cb);
builtin/fsck.c +11 -7
@@ -150,7 +150,7 @@ static int mark_object(struct object *obj, enum object_type type,return 0 ; obj->flags |=REACHABLE ; -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))/* * Further recursion does not need to be performed on this * object since it is a promisor object (so it does not need to
@@ -270,7 +270,7 @@ static void check_reachable_object(struct object *obj)* do a full fsck */ if (!(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */
@@ -391,7 +391,10 @@ static void check_connectivity(void)* traversal. */ for_each_loose_object (mark_loose_unreachable_referents,NULL ,0 ); -for_each_packed_object (mark_packed_unreachable_referents,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_unreachable_referents, +NULL , +0 ); }/* Look up all the requirements, warn about missing objects.. */
@@ -488,7 +491,7 @@ static void fsck_handle_reflog_oid(const char *refname, struct object_id *oid, refname, timestamp); obj->flags |=USED ;mark_object_reachable (obj); - }else if (!is_promisor_object (oid)) { + }else if (!is_promisor_object (the_repository, oid)) {error (_ ("%s: invalid reflog entry %s" ), refname,oid_to_hex (oid)); errors_found |=ERROR_REACHABLE ;
@@ -531,7 +534,7 @@ static int fsck_handle_ref(const char *refname, const char *referent UNUSED, con obj =parse_object (the_repository, oid);if (!obj) { -if (is_promisor_object (oid)) { +if (is_promisor_object (the_repository, oid)) {/* * Increment default_refs anyway, because this is a * valid ref.
@@ -966,7 +969,8 @@ int cmd_fsck(int argc,if (connectivity_only) {for_each_loose_object (mark_loose_for_connectivity,NULL ,0 ); -for_each_packed_object (mark_packed_for_connectivity,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_for_connectivity,NULL ,0 ); }else {prepare_alt_odb (the_repository);for (odb = the_repository->objects->odb; odb; odb = odb->next)
@@ -1011,7 +1015,7 @@ int cmd_fsck(int argc, &oid);if (!obj || !(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&oid)) +if (is_promisor_object (the_repository, &oid))continue ;error (_ ("%s: object missing" ),oid_to_hex (&oid)); errors_found |=ERROR_OBJECT ;
builtin/pack-objects.c +5 -2
@@ -3858,7 +3858,8 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,* Quietly ignore EXPECTED missing objects. This avoids problems with * staging them now and getting an odd error later. */ -if (!has_object (the_repository, &obj->oid,0 ) &&is_promisor_object (&obj->oid)) +if (!has_object (the_repository, &obj->oid,0 ) && +is_promisor_object (to_pack.repo, &obj->oid))return ;show_object (obj, name, data);
@@ -3927,7 +3928,9 @@ static int add_object_in_unpacked_pack(const struct object_id *oid,static void add_objects_in_unpacked_packs (void ) { - if (for_each_packed_object(add_object_in_unpacked_pack, NULL, + if (for_each_packed_object(to_pack.repo, + add_object_in_unpacked_pack, + NULL, FOR_EACH_OBJECT_PACK_ORDER | FOR_EACH_OBJECT_LOCAL_ONLY | FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS |
builtin/repack.c +1 -1
@@ -404,7 +404,7 @@ static void repack_promisor_objects(const struct pack_objects_args *args,* {type -> existing pack order} ordering when computing deltas instead * of a {type -> size} ordering, which may produce better deltas. */ -for_each_packed_object (write_oid, &cmd, +for_each_packed_object (the_repository, write_oid, &cmd,FOR_EACH_OBJECT_PROMISOR_ONLY );if (cmd.in ==-1 ) {
builtin/rev-list.c +1 -1
@@ -121,7 +121,7 @@ static inline void finish_object__ma(struct object *obj)return ;case MA_ALLOW_PROMISOR : -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;die ("unexpected missing %s object '%s'" ,type_name (obj->type),oid_to_hex (&obj->oid));
commit-graph.c +1 -1
@@ -1960,7 +1960,7 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx) ctx->progress =start_delayed_progress (_ ("Finding commits for commit graph among packed objects" ), ctx->approx_nr_objects); -for_each_packed_object (add_packed_commits, ctx, +for_each_packed_object (ctx->r, add_packed_commits, ctx,FOR_EACH_OBJECT_PACK_ORDER );if (ctx->progress_done < ctx->approx_nr_objects)display_progress (ctx->progress, ctx->approx_nr_objects);
fsck.c +1 -1
@@ -1295,7 +1295,7 @@ static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done, buf =repo_read_object_file (the_repository, oid, &type, &size);if (!buf) { -if (is_promisor_object (oid)) +if (is_promisor_object (the_repository, oid))continue ; ret |= report(options, oid, OBJ_BLOB, msg_missing,
list-objects.c +2 -2
@@ -75,7 +75,7 @@ static void process_blob(struct traversal_context *ctx,*/ if (ctx->revs->exclude_promisor_objects && !repo_has_object_file (the_repository, &obj->oid) && -is_promisor_object (&obj->oid)) +is_promisor_object (ctx->revs->repo, &obj->oid))return ; pathlen = path->len;
@@ -180,7 +180,7 @@ static void process_tree(struct traversal_context *ctx,* an incomplete list of missing objects. */ if (revs->exclude_promisor_objects && -is_promisor_object (&obj->oid)) +is_promisor_object (revs->repo, &obj->oid))return ;if (!revs->do_not_die_on_missing_objects)
object-store-ll.h +2 -2
@@ -550,7 +550,7 @@ typedef int each_packed_object_fn(const struct object_id *oid,int for_each_object_in_pack (struct packed_git *p,each_packed_object_fn ,void *data,enum for_each_object_flags flags); -int for_each_packed_object (each_packed_object_fn ,void *, -enum for_each_object_flags flags); +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags);#endif /* OBJECT_STORE_LL_H */
packfile.c +7 -7
@@ -2200,15 +2200,15 @@ int for_each_object_in_pack(struct packed_git *p,return r; } -int for_each_packed_object (each_packed_object_fn cb,void *data, -enum for_each_object_flags flags) +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags) {struct packed_git *p;int r =0 ;int pack_errors =0 ; -prepare_packed_git (the_repository); -for (p =get_all_packs (the_repository); p; p = p->next) { +prepare_packed_git (repo); +for (p =get_all_packs (repo); p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ; if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
@@ -2286,14 +2286,14 @@ static int add_promisor_object(const struct object_id *oid,return 0 ; } -int is_promisor_object (const struct object_id *oid) +int is_promisor_object (struct repository *r,const struct object_id *oid) {static struct oidset promisor_objects;static int promisor_objects_prepared;if (!promisor_objects_prepared) { -if (repo_has_promisor_remote (the_repository)) { -for_each_packed_object (add_promisor_object, +if (repo_has_promisor_remote (r)) { +for_each_packed_object (r, add_promisor_object, &promisor_objects,FOR_EACH_OBJECT_PROMISOR_ONLY |FOR_EACH_OBJECT_PACK_ORDER );
packfile.h +1 -1
@@ -201,7 +201,7 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,* Return 1 if an object in a promisor packfile is or refers to the given * object, 0 otherwise. */ -int is_promisor_object (const struct object_id *oid); +int is_promisor_object (struct repository *r,const struct object_id *oid);/* * Expose a function for fuzz testing.
promisor-remote.c +1 -1
@@ -283,7 +283,7 @@ void promisor_remote_get_direct(struct repository *repo, }for (i =0 ; i < remaining_nr; i++) { -if (is_promisor_object (&remaining_oids[i])) +if (is_promisor_object (repo, &remaining_oids[i]))die (_ ("could not fetch %s from promisor remote" ),oid_to_hex (&remaining_oids[i])); }
reachable.c +1 -1
@@ -324,7 +324,7 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,if (ignore_in_core_kept_packs) flags |=FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS ; - r =for_each_packed_object (add_recent_packed, &data, flags); + r =for_each_packed_object (revs->repo, add_recent_packed, &data, flags); done:oidset_clear (&data.extra_recent_oids);
revision.c +5 -4
@@ -390,7 +390,8 @@ static struct object *get_reference(struct rev_info *revs, const char *name,if (!object) {if (revs->ignore_missing)return NULL ; -if (revs->exclude_promisor_objects &&is_promisor_object (oid)) +if (revs->exclude_promisor_objects && +is_promisor_object (revs->repo, oid))return NULL ;if (revs->do_not_die_on_missing_objects) {oidset_insert (&revs->missing_commits, oid);
@@ -432,7 +433,7 @@ static struct commit *handle_commit(struct rev_info *revs,if (revs->ignore_missing_links || (flags &UNINTERESTING ))return NULL ;if (revs->exclude_promisor_objects && -is_promisor_object (&tag->tagged->oid)) +is_promisor_object (revs->repo, &tag->tagged->oid))return NULL ;if (revs->do_not_die_on_missing_objects && oid) {oidset_insert (&revs->missing_commits, oid);
@@ -1211,7 +1212,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit, revs->do_not_die_on_missing_objects;if (repo_parse_commit_gently (revs->repo, p, gently) <0 ) {if (revs->exclude_promisor_objects && -is_promisor_object (&p->object.oid)) { +is_promisor_object (revs->repo, &p->object.oid)) {if (revs->first_parent_only)break ;continue ;
@@ -3915,7 +3916,7 @@ int prepare_revision_walk(struct rev_info *revs) revs->treesame.name ="treesame" ;if (revs->exclude_promisor_objects) { -for_each_packed_object (mark_uninteresting, revs, +for_each_packed_object (revs->repo, mark_uninteresting, revs,FOR_EACH_OBJECT_PROMISOR_ONLY ); }
tag.c +1 -1
@@ -84,7 +84,7 @@ struct object *deref_tag(struct repository *r, struct object *o, const char *war o =NULL ; }if (!o && warn) { -if (last_oid &&is_promisor_object (last_oid)) +if (last_oid &&is_promisor_object (r, last_oid))return NULL ;if (!warnlen) warnlen =strlen (warn);
Signature · 1 line
2.47.0
PATCH v6 7/9 config: make `delta_base_cache_limit` a non-global variable
In reply to Karthik NayakThe delta_base_cache_limit variable is a global config variable used by multiple subsystems. Let's make this non-global, by adding this variable to the stack of each of the subsystems where it is used.
In gc.c we add it to the gc_config struct and also the constructor function. In index-pack.c we add it to the pack_idx_option struct and its constructor. Finally, in packfile.c we dynamically retrieve this value from the repository config, since the value is only used once in the entire subsystem.
These changes are made to remove the usage of delta_base_cache_limit as a global variable in packfile.c. This brings us one step closer to removing the USE_THE_REPOSITORY_VARIABLE definition in packfile.c which we complete in the next patch.
--- builtin/gc.c | 8 +++++++- builtin/index-pack.c | 10 +++++++--- config.c | 5 ----- environment.c | 1 - environment.h | 1 - pack-objects.h | 3 ++- pack-write.c | 1 + pack.h | 1 + packfile.c | 13 +++++++++++-- 9 files changed, 29 insertions(+), 14 deletions(-)
builtin/gc.c +7 -1
@@ -138,6 +138,7 @@ struct gc_config {char *repack_filter_to;unsigned long big_pack_threshold;unsigned long max_delta_cache_size; +size_t delta_base_cache_limit; };#define GC_CONFIG_INIT { \
@@ -153,6 +154,7 @@ struct gc_config { .prune_expire = xstrdup("2.weeks.ago"), \ .prune_worktrees_expire = xstrdup("3.months.ago"), \ .max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \ + .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \ }static void gc_config_release (struct gc_config *cfg)
@@ -168,6 +170,7 @@ static void gc_config(struct gc_config *cfg) {const char *value;char *owned =NULL ; +unsigned long longval;if (!git_config_get_value ("gc.packrefs" , &value)) {if (value && !strcmp (value,"notbare" ))
@@ -206,6 +209,9 @@ static void gc_config(struct gc_config *cfg)git_config_get_ulong ("gc.bigpackthreshold" , &cfg->big_pack_threshold);git_config_get_ulong ("pack.deltacachesize" , &cfg->max_delta_cache_size); +if (!git_config_get_ulong ("core.deltabasecachelimit" , &longval)) + cfg->delta_base_cache_limit = longval; +if (!git_config_get_string ("gc.repackfilter" , &owned)) {free (cfg->repack_filter); cfg->repack_filter = owned;
@@ -416,7 +422,7 @@ static uint64_t estimate_repack_memory(struct gc_config *cfg,* read_sha1_file() (either at delta calculation phase, or * writing phase) also fills up the delta base cache */ - heap += delta_base_cache_limit; + heap += cfg->delta_base_cache_limit;/* and of course pack-objects has its own delta cache */ heap += cfg->max_delta_cache_size;
builtin/index-pack.c +7 -3
@@ -1238,7 +1238,7 @@ static void parse_pack_objects(unsigned char *hash)* recursively checking if the resulting object is used as a base * for some more deltas. */ -static void resolve_deltas (void ) +static void resolve_deltas (struct pack_idx_option *opts) {int i;
@@ -1254,7 +1254,7 @@ static void resolve_deltas(void) nr_ref_deltas + nr_ofs_deltas); nr_dispatched =0 ; - base_cache_limit = delta_base_cache_limit * nr_threads; + base_cache_limit = opts->delta_base_cache_limit * nr_threads;if (nr_threads >1 ||getenv ("GIT_FORCE_THREADS" )) {init_thread ();work_lock ();
@@ -1604,6 +1604,10 @@ static int git_index_pack_config(const char *k, const char *v,else opts->flags &= ~WRITE_REV ; } +if (!strcmp (k,"core.deltabasecachelimit" )) { + opts->delta_base_cache_limit =git_config_ulong (k, v, ctx->kvi); +return 0 ; + }return git_default_config (k, v, ctx, cb); }
@@ -1930,7 +1934,7 @@ int cmd_index_pack(int argc,parse_pack_objects (pack_hash);if (report_end_of_input)write_in_full (2 ,"\0" ,1 ); -resolve_deltas (); +resolve_deltas (&opts);conclude_pack (fix_thin_pack, curr_pack, pack_hash);free (ofs_deltas);free (ref_deltas);
config.c +0 -5
@@ -1515,11 +1515,6 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.deltabasecachelimit" )) { - delta_base_cache_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -1
@@ -51,7 +51,6 @@ enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ;size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ;size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ; -size_t delta_base_cache_limit =96 *1024 *1024 ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
environment.h +0 -1
@@ -165,7 +165,6 @@ extern int zlib_compression_level;extern int pack_compression_level;extern size_t packed_git_window_size;extern size_t packed_git_limit; -extern size_t delta_base_cache_limit;extern unsigned long big_file_threshold;extern unsigned long pack_size_limit_cfg;extern int max_allowed_tree_depth;
pack-objects.h +2 -1
@@ -7,7 +7,8 @@struct repository ; -#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_BASE_CACHE_LIMIT (96 * 1024 * 1024)#define OE_DFS_STATE_BITS 2#define OE_DEPTH_BITS 12
pack-write.c +1 -0
@@ -21,6 +21,7 @@ void reset_pack_idx_option(struct pack_idx_option *opts)memset (opts,0 ,sizeof (*opts)); opts->version =2 ; opts->off32_limit =0x7fffffff ; + opts->delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; }static int sha1_compare (const void *_a,const void *_b)
pack.h +1 -0
@@ -58,6 +58,7 @@ struct pack_idx_option {*/ int anomaly_alloc, anomaly_nr;uint32_t *anomaly; +unsigned long delta_base_cache_limit; };void reset_pack_idx_option (struct pack_idx_option *);
packfile.c +11 -2
@@ -24,6 +24,8 @@#include "commit-graph.h" #include "pack-revindex.h" #include "promisor-remote.h" +#include "config.h" +#include "pack-objects.h" char *odb_pack_name (struct repository *r,struct strbuf *buf,const unsigned char *hash,const char *ext)
@@ -1496,7 +1498,9 @@ void clear_delta_base_cache(void) }static void add_delta_base_cache (struct packed_git *p,off_t base_offset, -void *base,unsigned long base_size,enum object_type type) +void *base,unsigned long base_size, +unsigned long delta_base_cache_limit, +enum object_type type) {struct delta_base_cache_entry *ent;struct list_head *lru, *tmp;
@@ -1697,6 +1701,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,struct unpack_entry_stack_ent *delta_stack = small_delta_stack;int delta_stack_nr =0 , delta_stack_alloc =UNPACK_ENTRY_STACK_PREALLOC ;int base_from_cache =0 ; +unsigned long delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; + +repo_config_get_ulong (r,"core.deltabasecachelimit" , &delta_base_cache_limit);write_pack_access_log (p, obj_offset);
@@ -1878,7 +1885,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,* before we are done using it. */ if (!external_base) -add_delta_base_cache (p, base_obj_offset, base, base_size, type); +add_delta_base_cache (p, base_obj_offset, base, + base_size, delta_base_cache_limit, + type);free (delta_data);free (external_base);
Signature · 1 line
2.47.0
PATCH v6 8/9 config: make `packed_git_(limit|window_size)` non-global variables
In reply to Karthik NayakThe variables packed_git_window_size and packed_git_limit are global config variables used in the packfile.c file. Since it is only used in this file, let's change it from being a global config variable to a local variable for the subsystem.
We do this by introducing a new local packfile_config struct in packfile.c and also adding the required function to parse the said config. We then use this within packfile.c to obtain the variables.
With this, we rid packfile.c from all global variable usage and this means we can also remove the USE_THE_REPOSITORY_VARIABLE guard from the file.
--- builtin/fast-import.c | 4 ++-- config.c | 17 ----------------- environment.c | 2 -- packfile.c | 23 +++++++++++++++-------- packfile.h | 2 +- repo-settings.c | 14 ++++++++++++++ repo-settings.h | 5 +++++ 7 files changed, 37 insertions(+), 30 deletions(-)
builtin/fast-import.c +2 -2
@@ -3539,7 +3539,7 @@ static void parse_argv(void)int cmd_fast_import (int argc,const char **argv,const char *prefix, - struct repository *repo UNUSED) +struct repository *repo) {unsigned int i;
@@ -3660,7 +3660,7 @@ int cmd_fast_import(int argc,fprintf (stderr," pools: %10lu KiB\n" , (unsigned long )((tree_entry_allocd + fi_mem_pool.pool_alloc) /1024 ));fprintf (stderr," objects: %10" PRIuMAX" KiB\n" , (alloc_count*sizeof (struct object_entry ))/1024 );fprintf (stderr,"---------------------------------------------------------------------\n" ); -pack_report (); +pack_report (repo);fprintf (stderr,"---------------------------------------------------------------------\n" );fprintf (stderr,"\n" ); }
config.c +0 -17
@@ -1493,28 +1493,11 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.packedgitwindowsize" )) { -int pgsz_x2 =getpagesize () *2 ; - packed_git_window_size =git_config_ulong (var, value, ctx->kvi); - -/* This value must be multiple of (pagesize * 2) */ - packed_git_window_size /= pgsz_x2; -if (packed_git_window_size <1 ) - packed_git_window_size =1 ; - packed_git_window_size *= pgsz_x2; -return 0 ; - } -if (!strcmp (var,"core.bigfilethreshold" )) { big_file_threshold =git_config_ulong (var, value, ctx->kvi);return 0 ; } -if (!strcmp (var,"core.packedgitlimit" )) { - packed_git_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -2
@@ -49,8 +49,6 @@ int fsync_object_files = -1;int use_fsync =-1 ;enum fsync_method fsync_method =FSYNC_METHOD_DEFAULT ;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ; -size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ; -size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
packfile.c +15 -8
@@ -1,4 +1,3 @@ -#define USE_THE_REPOSITORY_VARIABLE #include "git-compat-util.h" #include "environment.h"
@@ -48,15 +47,15 @@ static size_t pack_mapped;#define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; } -void pack_report (void ) +void pack_report (struct repository *repo) {fprintf (stderr,"pack_report: getpagesize() = %10" SZ_FMT "\n" "pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" "pack_report: core.packedGitLimit = %10" SZ_FMT "\n" ,sz_fmt (getpagesize ()), -sz_fmt (packed_git_window_size), -sz_fmt (packed_git_limit)); +sz_fmt (repo->settings.packed_git_window_size), +sz_fmt (repo->settings.packed_git_limit));fprintf (stderr,"pack_report: pack_used_ctr = %10u\n" "pack_report: pack_mmap_calls = %10u\n"
@@ -652,8 +651,15 @@ unsigned char *use_pack(struct packed_git *p,break ; }if (!win) { -size_t window_align = packed_git_window_size /2 ; +size_t window_align;off_t len; +struct repo_settings *settings; + +/* lazy load the settings incase it hasn't been setup */ +prepare_repo_settings (p->repo); + settings = &p->repo->settings; + + window_align = settings->packed_git_window_size /2 ;if (p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name);
@@ -661,11 +667,12 @@ unsigned char *use_pack(struct packed_git *p,CALLOC_ARRAY (win,1 ); win->offset = (offset / window_align) * window_align; len = p->pack_size - win->offset; -if (len > packed_git_window_size) - len = packed_git_window_size; +if (len > settings->packed_git_window_size) + len = settings->packed_git_window_size; win->len = (size_t )len; pack_mapped += win->len; -while (packed_git_limit < pack_mapped + +while (settings->packed_git_limit < pack_mapped &&unuse_one_window (p)) ;/* nothing */ win->base = xmmap_gently(NULL, win->len,
packfile.h +1 -1
@@ -89,7 +89,7 @@ unsigned long repo_approximate_object_count(struct repository *r);struct packed_git *find_oid_pack (const struct object_id *oid,struct packed_git *packs); -void pack_report (void ); +void pack_report (struct repository *repo);/* * mmap the index file for the specified packfile (if it is not
repo-settings.c +14 -0
@@ -26,6 +26,7 @@ void prepare_repo_settings(struct repository *r)const char *strval;int manyfiles;int read_changed_paths; +unsigned long longval;if (!r->gitdir)BUG ("Cannot add settings for uninitialized repository" );
@@ -123,6 +124,19 @@ void prepare_repo_settings(struct repository *r)* removed. */ r->settings.command_requires_full_index =1 ; + +if (!repo_config_get_ulong (r,"core.packedgitwindowsize" , &longval)) { +int pgsz_x2 =getpagesize () *2 ; + +/* This value must be multiple of (pagesize * 2) */ + longval /= pgsz_x2; +if (longval <1 ) + longval =1 ; + r->settings.packed_git_window_size = longval * pgsz_x2; + } + +if (!repo_config_get_ulong (r,"core.packedgitlimit" , &longval)) + r->settings.packed_git_limit = longval; }enum log_refs_config repo_settings_get_log_all_ref_updates (struct repository *repo)
repo-settings.h +5 -0
@@ -57,12 +57,17 @@ struct repo_settings {int core_multi_pack_index;int warn_ambiguous_refs;/* lazily loaded via accessor */ + +size_t packed_git_window_size; +size_t packed_git_limit; };#define REPO_SETTINGS_INIT { \ .index_version = -1, \ .core_untracked_cache = UNTRACKED_CACHE_KEEP, \ .fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \ .warn_ambiguous_refs = -1, \ + .packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE, \ + .packed_git_limit = DEFAULT_PACKED_GIT_LIMIT, \ }void prepare_repo_settings (struct repository *r);
Signature · 1 line
2.47.0
Re: [PATCH v6 8/9] config: make `packed_git_(limit|window_size)` non-global variables
Quoted 8 lines · Karthik Nayak
The variables
packed_git_window_sizeandpacked_git_limitare global config variables used in thepackfile.cfile. Since it is only used in this file, let's change it from being a global config variable to a local variable for the subsystem.We do this by introducing a new local
packfile_configstruct inpackfile.cand also adding the required function to parse the said config. We then use this withinpackfile.cto obtain the variables.
This patch has no string "packfile_config" in it, other than the one in the above string. A stale description?
packfile.c unsigned char *use_pack(struct packed_git *p, · line 658
Quoted 7 lines
packfile.c +4
-1if (!win) { -size_t window_align = packed_git_window_size /2 ; +size_t window_align;off_t len; +struct repo_settings *settings; + +/* lazy load the settings incase it hasn't been setup */
"incase" -> "in case"?
packfile.c unsigned char *use_pack(struct packed_git *p, · line 660
Quoted 2 lines
packfile.c +2
-0+prepare_repo_settings (p->repo); + settings = &p->repo->settings;
This change is curious. How can p->repo be uninitialized? p is a packed-git list created in some repository, surely it should already be initialized, no?
Quoted 10 lines
packfile.c +2
-2+ + window_align = settings->packed_git_window_size /2 ;if (p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name);@@ -661,11 +667,12 @@ unsigned char *use_pack(struct packed_git *p,CALLOC_ARRAY (win,1 ); win->offset = (offset / window_align) * window_align; len = p->pack_size - win->offset; -if (len > packed_git_window_size) - len = packed_git_window_size;
packfile.c unsigned char *use_pack(struct packed_git *p, · line 678
Quoted 10 lines
packfile.c +4
-1+if (len > settings->packed_git_window_size) + len = settings->packed_git_window_size; win->len = (size_t )len; pack_mapped += win->len; -while (packed_git_limit < pack_mapped + +while (settings->packed_git_limit < pack_mapped &&unuse_one_window (p)) ;/* nothing */ win->base = xmmap_gently(NULL, win->len,
Other than that, the changes to the above block that uses the local variable "settings" looks good.
Thanks.
Re: [PATCH v6 8/9] config: make `packed_git_(limit|window_size)` non-global variables
Quoted 7 lines · Junio C Hamano
Quoted 5 lines · Karthik Nayak
The variables
packed_git_window_sizeandpacked_git_limitare global config variables used in thepackfile.cfile. Since it is only used in this file, let's change it from being a global config variable to a local variable for the subsystem.
Quoted 7 lines
Quoted 3 lines
We do this by introducing a new local
packfile_configstruct inpackfile.cand also adding the required function to parse the said config. We then use this withinpackfile.cto obtain the variables.This patch has no string "
packfile_config" in it, other than the one in the above string. A stale description?
Yup indeed.
Quoted 10 lines
packfile.c unsigned char *use_pack(struct packed_git *p, · line 658
Quoted 7 lines
packfile.c +4
-1if (!win) { -size_t window_align = packed_git_window_size /2 ; +size_t window_align;off_t len; +struct repo_settings *settings; + +/* lazy load the settings incase it hasn't been setup */ "incase" -> "in case"?
Will change.
Quoted 7 lines
packfile.c unsigned char *use_pack(struct packed_git *p, · line 660
Quoted 2 lines
packfile.c +2
-0+prepare_repo_settings (p->repo); + settings = &p->repo->settings;This change is curious. How can
p->repobe uninitialized? p is a packed-git list created in some repository, surely it should already be initialized, no?
Here p->repo itself is expected to be initialized. We're however trying to initialize p->repo->settings. Which might not have been. If it is, prepare_repo_settings will return early.
Quoted 26 lines
Quoted 11 lines
packfile.c +2
-2+ + window_align = settings->packed_git_window_size /2 ;if (p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name);@@ -661,11 +667,12 @@ unsigned char *use_pack(struct packed_git *p,CALLOC_ARRAY (win,1 ); win->offset = (offset / window_align) * window_align; len = p->pack_size - win->offset; -if (len > packed_git_window_size) - len = packed_git_window_size;packfile.c unsigned char *use_pack(struct packed_git *p, · line 678
Quoted 10 lines
packfile.c +4
-1+if (len > settings->packed_git_window_size) + len = settings->packed_git_window_size; win->len = (size_t )len; pack_mapped += win->len; -while (packed_git_limit < pack_mapped + +while (settings->packed_git_limit < pack_mapped &&unuse_one_window (p)) ;/* nothing */ win->base = xmmap_gently(NULL, win->len,Other than that, the changes to the above block that uses the local variable "settings" looks good.
Thanks.
PATCH v6 9/9 midx: add repository to `multi_pack_index` struct
In reply to Karthik NayakThe multi_pack_index struct represents the MIDX for a repository. Here, we add a pointer to the repository in this struct, allowing direct use of the repository variable without relying on the global the_repository struct.
With this addition, we can determine the repository associated with a bitmap_index struct. A bitmap_index points to either a packed_git or a multi_pack_index, both of which have direct repository references. To support this, we introduce a static helper function, bitmap_repo, in pack-bitmap.c, which retrieves a repository given a bitmap_index.
With this, we clear up all usages of the_repository within pack-bitmap.c and also remove the USE_THE_REPOSITORY_VARIABLE definition. Bringing us another step closer to remove all global variable usage.
Although this change also opens up the potential to clean up midx.c, doing so would require additional refactoring to pass the repository struct to functions where the MIDX struct is created: a task better suited for future patches.
--- midx.c | 1 + midx.h | 3 ++ pack-bitmap.c | 90 +++++++++++++++++++++++++++++++-------------------- 3 files changed, 59 insertions(+), 35 deletions(-)
midx.c +1 -0
@@ -131,6 +131,7 @@ static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir m->data = midx_map; m->data_len = midx_size; m->local = local; + m->repo = the_repository; m->signature =get_be32 (m->data);if (m->signature !=MIDX_SIGNATURE )
midx.h +3 -0
@@ -71,6 +71,9 @@ struct multi_pack_index {const char **pack_names;struct packed_git **packs; + +struct repository *repo; +char object_dir[FLEX_ARRAY ]; };
pack-bitmap.c +55 -35
251 lines not on this page · raw message
Signature · 1 line
2.47.0
Quoted 4 lines · Karthik Nayak
Changes in v6:
- Lazy load repository settings inpackfile.c. This ensures that the settings are available for sure and we do not rely on callees setting it.
- Usesize_tfordelta_base_cache_limit.
I'll trust the reviews made while I was gone and will comment only on the differences between the last iteration.
builtin/gc.c struct gc_config { · line 142
Quoted 11 lines
builtin/gc.c +1
-1@@ -138,7 +138,7 @@ struct gc_config {char *repack_filter_to;unsigned long big_pack_threshold;unsigned long max_delta_cache_size; -unsigned long delta_base_cache_limit; +size_t delta_base_cache_limit; };
Makes sense.
Quoted 9 lines
builtin/gc.c +1
-0@@ -170,6 +170,7 @@ static void gc_config(struct gc_config *cfg) {const char *value;char *owned =NULL ; +unsigned long longval;if (!git_config_get_value ("gc.packrefs" , &value)) {if (value && !strcmp (value,"notbare" ))
builtin/gc.c static void gc_config(struct gc_config *cfg) · line 213
Quoted 7 lines
builtin/gc.c +3
-1@@ -207,7 +208,9 @@ static void gc_config(struct gc_config *cfg)git_config_get_ulong ("gc.bigpackthreshold" , &cfg->big_pack_threshold);git_config_get_ulong ("pack.deltacachesize" , &cfg->max_delta_cache_size); -git_config_get_ulong ("core.deltabasecachelimit" , &cfg->delta_base_cache_limit); + +if (!git_config_get_ulong ("core.deltabasecachelimit" , &longval)) + cfg->delta_base_cache_limit = longval;
And this is a sensible way to fill size_t member with the value read into a ulong. Should "longval" be named after "unsigned long" instead of "long", by the way?
There is a required SP missing inside "if(!".
Quoted 5 lines
packfile.c +0
-0
packfile.c unsigned char *use_pack(struct packed_git *p, · line 660
Quoted 9 lines
packfile.c +5
-1@@ -653,7 +653,11 @@ unsigned char *use_pack(struct packed_git *p,if (!win) {size_t window_align;off_t len; -struct repo_settings *settings = &p->repo->settings; +struct repo_settings *settings; + +/* lazy load the settings incase it hasn't been setup */ +prepare_repo_settings (p->repo); + settings = &p->repo->settings;
This is a bit curious. I'll read the individual patch that has this change before commenting on it.
Quoted 45 lines · Junio C Hamano
Quoted 4 lines · Karthik Nayak
Changes in v6:
- Lazy load repository settings inpackfile.c. This ensures that the settings are available for sure and we do not rely on callees setting it.
- Usesize_tfordelta_base_cache_limit.I'll trust the reviews made while I was gone and will comment only on the differences between the last iteration.
builtin/gc.c struct gc_config { · line 142
Quoted 11 lines
builtin/gc.c +1
-1@@ -138,7 +138,7 @@ struct gc_config {char *repack_filter_to;unsigned long big_pack_threshold;unsigned long max_delta_cache_size; -unsigned long delta_base_cache_limit; +size_t delta_base_cache_limit; };Makes sense.
Quoted 9 lines
builtin/gc.c +1
-0@@ -170,6 +170,7 @@ static void gc_config(struct gc_config *cfg) {const char *value;char *owned =NULL ; +unsigned long longval;if (!git_config_get_value ("gc.packrefs" , &value)) {if (value && !strcmp (value,"notbare" ))builtin/gc.c static void gc_config(struct gc_config *cfg) · line 213
Quoted 7 lines
builtin/gc.c +3
-1@@ -207,7 +208,9 @@ static void gc_config(struct gc_config *cfg)git_config_get_ulong ("gc.bigpackthreshold" , &cfg->big_pack_threshold);git_config_get_ulong ("pack.deltacachesize" , &cfg->max_delta_cache_size); -git_config_get_ulong ("core.deltabasecachelimit" , &cfg->delta_base_cache_limit); + +if (!git_config_get_ulong ("core.deltabasecachelimit" , &longval)) + cfg->delta_base_cache_limit = longval;And this is a sensible way to fill
size_tmember with the value read into a ulong. Should "longval" be named after "unsigned long" instead of "long", by the way?
Quoted 2 lines
There is a required SP missing inside "if(!".
Agreed, will fix both and send in a new version.
Quoted 17 lines
Quoted 5 lines
packfile.c +0
-0packfile.c unsigned char *use_pack(struct packed_git *p, · line 660
Quoted 9 lines
packfile.c +5
-1@@ -653,7 +653,11 @@ unsigned char *use_pack(struct packed_git *p,if (!win) {size_t window_align;off_t len; -struct repo_settings *settings = &p->repo->settings; +struct repo_settings *settings; + +/* lazy load the settings incase it hasn't been setup */ +prepare_repo_settings (p->repo); + settings = &p->repo->settings;This is a bit curious. I'll read the individual patch that has this change before commenting on it.
v7 cover packfile: avoid using the 'the_repository' global variable
In reply to Karthik NayakThe packfile.c file uses the global variable 'the_repository' extensively throughout the code. Let's remove all usecases of this, by modifying the required functions to accept a 'struct repository' instead. This is to clean up usage of global state.
The first 3 patches are mostly internal to packfile.c, we add the repository field to the packed_git struct and this is used to clear up some useages of the global variables.
The next 3 patches are more disruptive, they modify the function definition of odb_pack_name, has_object[_kept]_pack and for_each_packed_object to receive a repository, helping remove other usages of 'the_repository' variable.
Finally, the last two patches deal with global config values. These values are localized.
For v5 onwards, I've rebased the series off the new master: 8f8d6eee53 (The seventh batch, 2024-11-01), as a dependency for this series 'jk/dumb-http-finalize' was merged to master. I've found no conflicts while merging with seen & next. But since this series does touch multiple files, there could be future conflicts.
Changes in v7:
- Cleanup stale commit message.
- Add missing space in if statement.
- Fix typo s/incase/in case/.
Changes in v6:
- Lazy load repository settings in packfile.c. This ensures that the settings are available for sure and we do not rely on callees setting it.
- Use size_t for delta_base_cache_limit.
Changes in v5:
- Move packed_git* settings to repo_settings to ensure we don't keep reparsing the settings in use_pack.
Changes in v4:
- Renamed the repository field within packed_git and multi_pack_index from r to repo, while keeping function parameters to be r.
- Fixed bad braces.
Changes in v3:
- Improved commit messages. In the first commit to talk about how packed_git struct could also be part of the alternates of a repository. In the 7th commit to talk about the motive behind removing the global variable.
- Changed 'packed_git->repo' to 'packed_git->r' to keep it consistent with the rest of the code base.
- Replaced 'the_repository' with locally available access to the repository struct in multiple regions.
- Removed unecessary inclusion of the 'repository.h' header file by forward declaring the 'repository' struct.
- Replace memcpy with hashcpy.
- Change the logic in the 7th patch to use if else statements.
- Added an extra commit to cleanup pack-bitmap.c.
Karthik Nayak (9):
packfile: add repository to struct packed_git
packfile: use repository from packed_git directly
packfile: pass repository to static function in the file
packfile: pass down repository to odb_pack_name
packfile: pass down repository to has_object[_kept]_pack
packfile: pass down repository to for_each_packed_object
config: make delta_base_cache_limit a non-global variable
config: make packed_git_(limit|window_size) non-global variables
midx: add repository to multi_pack_index struct
builtin/cat-file.c | 7 +- builtin/count-objects.c | 2 +- builtin/fast-import.c | 15 ++-- builtin/fsck.c | 20 +++--- builtin/gc.c | 8 ++- builtin/index-pack.c | 20 ++++-- builtin/pack-objects.c | 11 +-- builtin/pack-redundant.c | 2 +- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 4 +- config.c | 22 ------ connected.c | 3 +- diff.c | 3 +- environment.c | 3 - environment.h | 1 - fsck.c | 2 +- http.c | 4 +- list-objects.c | 7 +- midx-write.c | 2 +- midx.c | 3 +- midx.h | 3 + object-store-ll.h | 9 ++- pack-bitmap.c | 90 +++++++++++++++--------- pack-objects.h | 3 +- pack-write.c | 1 + pack.h | 1 + packfile.c | 148 +++++++++++++++++++++++---------------- packfile.h | 18 +++-- promisor-remote.c | 2 +- prune-packed.c | 2 +- reachable.c | 4 +- repo-settings.c | 14 ++++ repo-settings.h | 5 ++ revision.c | 13 ++-- tag.c | 2 +- 36 files changed, 268 insertions(+), 190 deletions(-)
Range-diff against v6:
-: ---------- > 1: 6c00e25c86 packfile: add repository to struct `packed_git`
-: ---------- > 2: 70fc8a79af packfile: use `repository` from `packed_git` directly
-: ---------- > 3: 167a1f3a11 packfile: pass `repository` to static function in the file
-: ---------- > 4: b7cfe78217 packfile: pass down repository to `odb_pack_name`
-: ---------- > 5: 5566f5554c packfile: pass down repository to `has_object[_kept]_pack`
-: ---------- > 6: 1b26e45a9b packfile: pass down repository to `for_each_packed_object`
1: 89313cfed4 ! 7: 1bdc34f4d8 config: make `delta_base_cache_limit` a non-global variable
@@ builtin/gc.c: static void gc_config(struct gc_config *cfg)
{
const char *value;
char *owned = NULL;
-+ unsigned long longval;
++ unsigned long ulongval;
if (!git_config_get_value("gc.packrefs", &value)) {
if (value && !strcmp(value, "notbare"))
@@ builtin/gc.c: static void gc_config(struct gc_config *cfg)
git_config_get_ulong("gc.bigpackthreshold", &cfg->big_pack_threshold);
git_config_get_ulong("pack.deltacachesize", &cfg->max_delta_cache_size);
-+ if(!git_config_get_ulong("core.deltabasecachelimit", &longval))
-+ cfg->delta_base_cache_limit = longval;
++ if (!git_config_get_ulong("core.deltabasecachelimit", &ulongval))
++ cfg->delta_base_cache_limit = ulongval;
+
if (!git_config_get_string("gc.repackfilter", &owned)) {
free(cfg->repack_filter); 2: 3a8e3b88df ! 8: 7b6baa89ac config: make packed_git_(limit|window_size) non-global variables
@@ Commit message
this file, let's change it from being a global config variable to a
local variable for the subsystem.
- We do this by introducing a new local packfile_config struct in
- packfile.c and also adding the required function to parse the said
- config. We then use this within packfile.c to obtain the variables.
-
With this, we rid packfile.c from all global variable usage and this means we can also remove the USE_THE_REPOSITORY_VARIABLE guard from the file.
@@ packfile.c: unsigned char *use_pack(struct packed_git *p,
off_t len;
+ struct repo_settings *settings;
+
-+ /* lazy load the settings incase it hasn't been setup */
++ /* lazy load the settings in case it hasn't been setup */
+ prepare_repo_settings(p->repo);
+ settings = &p->repo->settings;
+
3: 2f9a146978 = 9: a3667d87ec midx: add repository to multi_pack_index struct
Signature · 1 line
2.47.0
PATCH v7 1/9 packfile: add repository to struct `packed_git`
The struct packed_git holds information regarding a packed object file. Let's add the repository variable to this object, to represent the repository that this packfile belongs to. This helps remove dependency on the global the_repository object in packfile.c by simply using repository information now readily available in the struct.
We do need to consider that a pack file could be part of the alternates of a repository, but considering that we only have one repository struct and also that we currently anyways use 'the_repository'. We should be OK with this change.
We also modify alloc_packed_git to ensure that the repository is added to newly created packed_git structs. This requires modifying the function and all its callee to pass the repository object down the levels.
--- builtin/fast-import.c | 3 ++- builtin/index-pack.c | 6 ++++-- commit-graph.c | 2 +- connected.c | 3 ++- http.c | 2 +- midx-write.c | 2 +- midx.c | 2 +- object-store-ll.h | 5 +++++ packfile.c | 15 +++++++++------ packfile.h | 6 ++++-- 10 files changed, 30 insertions(+), 16 deletions(-)
builtin/fast-import.c +2 -1
@@ -765,6 +765,7 @@ static void start_packfile(void) p->pack_fd = pack_fd; p->do_not_close =1 ; + p->repo = the_repository; pack_file =hashfd (pack_fd, p->pack_name); pack_data = p;
@@ -888,7 +889,7 @@ static void end_packfile(void) idx_name =keep_pack (create_index ());/* Register the packfile with core git's machinery. */ - new_p =add_packed_git (idx_name,strlen (idx_name),1 ); + new_p =add_packed_git (pack_data->repo, idx_name,strlen (idx_name),1 );if (!new_p)die ("core git rejected index %s" , idx_name); all_packs[pack_id] = new_p;
builtin/index-pack.c +4 -2
@@ -1552,7 +1552,8 @@ static void final(const char *final_pack_name, const char *curr_pack_name,if (do_fsck_object) {struct packed_git *p; - p =add_packed_git (final_index_name,strlen (final_index_name),0 ); + p =add_packed_git (the_repository, final_index_name, +strlen (final_index_name),0 );if (p)install_packed_git (the_repository, p); }
@@ -1650,7 +1651,8 @@ static void read_v2_anomalous_offsets(struct packed_git *p,static void read_idx_option (struct pack_idx_option *opts,const char *pack_name) { -struct packed_git *p =add_packed_git (pack_name,strlen (pack_name),1 ); +struct packed_git *p =add_packed_git (the_repository, pack_name, +strlen (pack_name),1 );if (!p)die (_ ("Cannot open existing pack file '%s'" ), pack_name);
commit-graph.c +1 -1
@@ -1914,7 +1914,7 @@ static int fill_oids_from_packs(struct write_commit_graph_context *ctx,struct packed_git *p;strbuf_setlen (&packname, dirlen);strbuf_addstr (&packname, pack_indexes->items[i].string); - p =add_packed_git (packname.buf, packname.len,1 ); + p =add_packed_git (ctx->r, packname.buf, packname.len,1 );if (!p) { ret =error (_ ("error adding pack %s" ), packname.buf); goto cleanup;
connected.c +2 -1
@@ -54,7 +54,8 @@ int check_connected(oid_iterate_fn fn, void *cb_data,strbuf_add (&idx_file, transport->pack_lockfiles.items[0 ].string, base_len);strbuf_addstr (&idx_file,".idx" ); - new_pack =add_packed_git (idx_file.buf, idx_file.len,1 ); + new_pack =add_packed_git (the_repository, idx_file.buf, + idx_file.len,1 );strbuf_release (&idx_file); }
http.c +1 -1
@@ -2439,7 +2439,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,if (!tmp_idx)return -1 ; - new_pack =parse_pack_index (sha1, tmp_idx); + new_pack =parse_pack_index (the_repository, sha1, tmp_idx);if (!new_pack) {unlink (tmp_idx);free (tmp_idx);
midx-write.c +1 -1
@@ -154,7 +154,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,return ;ALLOC_GROW (ctx->info, ctx->nr +1 , ctx->alloc); - p =add_packed_git (full_path, full_path_len,0 ); + p =add_packed_git (the_repository, full_path, full_path_len,0 );if (!p) {warning (_ ("failed to add packfile '%s'" ), full_path);
midx.c +1 -1
@@ -464,7 +464,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,strhash (key.buf), key.buf, struct packed_git, packmap_ent);if (!p) { - p =add_packed_git (pack_name.buf, pack_name.len, m->local); + p =add_packed_git (r, pack_name.buf, pack_name.len, m->local);if (p) {install_packed_git (r, p);list_add_tail (&p->mru, &r->objects->packed_git_mru);
object-store-ll.h +5 -0
@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;struct object_directory {struct object_directory *next;
@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo dentoes the repository this packed file belongs to */ +struct repository *repo; +/* something like ".git/objects/pack/xxxxx.pack" */ char pack_name[FLEX_ARRAY ];/* more */ };
packfile.c +9 -6
@@ -217,11 +217,12 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)return ntohl (level1_ofs[value]); } -static struct packed_git *alloc_packed_git (int extra) +static struct packed_git *alloc_packed_git (struct repository *r,int extra) {struct packed_git *p =xmalloc (st_add (sizeof (*p), extra));memset (p,0 ,sizeof (*p)); p->pack_fd =-1 ; + p->repo = r;return p; }
@@ -233,11 +234,12 @@ static char *pack_path_from_idx(const char *idx_path)return xstrfmt ("%.*s.pack" , (int )len, idx_path); } -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path) {char *path =pack_path_from_idx (idx_path);size_t alloc =st_add (strlen (path),1 ); -struct packed_git *p =alloc_packed_git (alloc); +struct packed_git *p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path);
@@ -703,7 +705,8 @@ void unuse_pack(struct pack_window **w_cursor) } } -struct packed_git *add_packed_git (const char *path,size_t path_len,int local) +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local) {struct stat st;size_t alloc;
@@ -721,7 +724,7 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)* the use xsnprintf double-checks that) */ alloc =st_add3 (path_len,strlen (".promisor" ),1 ); - p =alloc_packed_git (alloc); + p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, path_len);xsnprintf (p->pack_name + path_len, alloc - path_len,".keep" );
@@ -877,7 +880,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,/* Don't reopen a pack we already have. */ if (!hashmap_get (&data->r->objects->pack_map, &hent, pack_name)) { - p =add_packed_git (full_name, full_name_len, data->local); + p =add_packed_git (data->r, full_name, full_name_len, data->local);if (p)install_packed_git (data->r, p); }
packfile.h +4 -2
@@ -46,7 +46,8 @@ const char *pack_basename(struct packed_git *p);* and does not add the resulting packed_git struct to the internal list of * packs. You probably want add_packed_git() instead. */ -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path);typedef void each_file_in_pack_dir_fn (const char *full_path,size_t full_path_len,const char *file_name,void *data);
@@ -113,7 +114,8 @@ void close_pack(struct packed_git *);void close_object_store (struct raw_object_store *o);void unuse_pack (struct pack_window **);void clear_delta_base_cache (void ); -struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local);/* * Unlink the .pack and associated extension files.
Signature · 1 line
2.47.0
Re: [PATCH v7 1/9] packfile: add repository to struct `packed_git`
Quoted 15 lines · Karthik Nayak
[snip]
object-store-ll.h +1
-0@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;struct object_directory {struct object_directory *next;
object-store-ll.h struct packed_git { · line 140
Quoted 5 lines
object-store-ll.h +2
-0@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo dentoes the repository this packed file belongs to */
Small typo here, I think you mean "denotes".
That's all I've got about all other changes in this patch series.
--
Toon
Re: [PATCH v7 1/9] packfile: add repository to struct `packed_git`
Quoted 14 lines · Toon Claes
Quoted 12 lines · Karthik Nayak
[snip]
object-store-ll.h +1
-0@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;
Quoted 11 lines
object-store-ll.h struct packed_git { · line 140
Quoted 8 lines
object-store-ll.h +2
-0struct object_directory {struct object_directory *next;@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo dentoes the repository this packed file belongs to */ Small typo here, I think you mean "denotes".
Indeed. I'll add it in locally but will avoid a re-roll just for the typo.
Quoted 2 lines
That's all I've got about all other changes in this patch series.
Thanks for the review!
- Karthik
Re: [PATCH v7 1/9] packfile: add repository to struct `packed_git`
Quoted 2 lines · karthik nayak
Indeed. I'll add it in locally but will avoid a re-roll just for the typo.
FWIW, I'd _strongly_ prefer people *not* phrase it that way.
It is very much sensible to refrain sending an another reroll immediately only to correct a small typo. IOW, "avoid a" -> "avoid an immediate" would be very much appreciated.
But in the end, the final version should not waste all the work that went into reviewing the series.
Thanks.
Re: [PATCH v7 1/9] packfile: add repository to struct `packed_git`
Quoted 5 lines · Junio C Hamano
Quoted 2 lines · karthik nayak
Indeed. I'll add it in locally but will avoid a re-roll just for the typo.
Quoted 9 lines
FWIW, I'd _strongly_ prefer people *not* phrase it that way.
It is very much sensible to refrain sending an another reroll immediately only to correct a small typo. IOW, "avoid a" -> "avoid an immediate" would be very much appreciated.
But in the end, the final version should not waste all the work that went into reviewing the series.
Thanks for pointing it out, I agree and didn't really consider how that seems. I will be more mindful!
Quoted 1 line
Thanks.
Re: [PATCH v7 1/9] packfile: add repository to struct `packed_git`
In reply to Toon ClaesQuoted 14 lines · Toon Claes
Quoted 12 lines · Karthik Nayak
[snip]
object-store-ll.h +1
-0@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;
Quoted 10 lines
object-store-ll.h struct packed_git { · line 140
Quoted 8 lines
object-store-ll.h +2
-0struct object_directory {struct object_directory *next;@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo dentoes the repository this packed file belongs to */ Small typo here, I think you mean "denotes".
Likewise for "packed file", which should be "packfile".
Thanks,
Taylor
Re: [PATCH v7 1/9] packfile: add repository to struct `packed_git`
Quoted 26 lines · Taylor Blau
Quoted 14 lines · Toon Claes
Quoted 12 lines · Karthik Nayak
[snip]
object-store-ll.h +1
-0@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;Quoted 10 lines
object-store-ll.h struct packed_git { · line 140
Quoted 8 lines
object-store-ll.h +2
-0struct object_directory {struct object_directory *next;@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo dentoes the repository this packed file belongs to */ Small typo here, I think you mean "denotes".
Quoted 1 line
Likewise for "packed file", which should be "packfile".
Indeed, will fix both in the next version.
PATCH v7 2/9 packfile: use `repository` from `packed_git` directly
In reply to Karthik NayakIn the previous commit, we introduced the repository structure inside packed_git. This provides an alternative route instead of using the global the_repository variable. Let's modify packfile.c now to use this field wherever possible instead of relying on the global state. There are still a few instances of the_repository usage in the file, where there is no struct packed_git locally available, which will be fixed in the following commits.
--- packfile.c | 50 +++++++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 23 deletions(-)
packfile.c +27 -23
@@ -79,7 +79,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)size_t idx_size;int fd =git_open (path), ret;struct stat st; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (fd <0 )return -1 ;
@@ -243,7 +243,7 @@ struct packed_git *parse_pack_index(struct repository *r, unsigned char *sha1,memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path); -hashcpy (p->hash, sha1, the_repository->hash_algo); +hashcpy (p->hash, sha1, p->repo->hash_algo);if (check_packed_git_idx (idx_path, p)) {free (p);return NULL ;
@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = the_repository->objects->packed_git; p; p = p->next) +for (p = current->repo->objects->packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -540,7 +540,7 @@ static int open_packed_git_1(struct packed_git *p)unsigned char hash[GIT_MAX_RAWSZ ];unsigned char *idx_hash;ssize_t read_result; -const unsigned hashsz = the_hash_algo->rawsz; +const unsigned hashsz = p->repo->hash_algo->rawsz;if (open_pack_index (p))return error ("packfile %s index unavailable" , p->pack_name);
@@ -597,7 +597,7 @@ static int open_packed_git_1(struct packed_git *p)if (read_result != hashsz)return error ("packfile %s signature is unavailable" , p->pack_name); idx_hash = ((unsigned char *)p->index_data) + p->index_size - hashsz *2 ; -if (!hasheq (hash, idx_hash, the_repository->hash_algo)) +if (!hasheq (hash, idx_hash, p->repo->hash_algo))return error ("packfile %s does not match index" , p->pack_name);return 0 ; }
@@ -637,7 +637,7 @@ unsigned char *use_pack(struct packed_git *p,*/ if (!p->pack_size && p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name); -if (offset > (p->pack_size - the_hash_algo->rawsz)) +if (offset > (p->pack_size - p->repo->hash_algo->rawsz))die ("offset beyond end of packfile (truncated pack?)" );if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" ));
@@ -711,6 +711,7 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,struct stat st;size_t alloc;struct packed_git *p; +struct object_id oid;/* * Make sure a corresponding .pack file exists and that
@@ -751,9 +752,13 @@ struct packed_git *add_packed_git(struct repository *r, const char *path, p->pack_size = st.st_size; p->pack_local = local; p->mtime = st.st_mtime; -if (path_len < the_hash_algo->hexsz || -get_hash_hex (path + path_len - the_hash_algo->hexsz, p->hash)) -hashclr (p->hash, the_repository->hash_algo); +if (path_len < r->hash_algo->hexsz || +get_oid_hex_algop (path + path_len - r->hash_algo->hexsz, &oid, + r->hash_algo)) +hashclr (p->hash, r->hash_algo); +else +hashcpy (p->hash, oid.hash, r->hash_algo); +return p; }
@@ -1243,9 +1248,9 @@ off_t get_delta_base(struct packed_git *p, }else if (type ==OBJ_REF_DELTA ) {/* The base entry _must_ be in the same pack */ struct object_id oid; -oidread (&oid, base_info, the_repository->hash_algo); +oidread (&oid, base_info, p->repo->hash_algo); base_offset =find_pack_entry_one (&oid, p); - *curpos += the_hash_algo->rawsz; + *curpos += p->repo->hash_algo->rawsz; }else die ("I am totally screwed" );return base_offset;
@@ -1266,7 +1271,7 @@ static int get_delta_base_oid(struct packed_git *p, {if (type ==OBJ_REF_DELTA ) {unsigned char *base =use_pack (p, w_curs, curpos,NULL ); -oidread (oid, base, the_repository->hash_algo); +oidread (oid, base, p->repo->hash_algo);return 0 ; }else if (type ==OBJ_OFS_DELTA ) {uint32_t base_pos;
@@ -1608,7 +1613,7 @@ int packed_object_info(struct repository *r, struct packed_git *p, goto out; } }else -oidclr (oi->delta_base_oid, the_repository->hash_algo); +oidclr (oi->delta_base_oid, p->repo->hash_algo); } oi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :
@@ -1897,7 +1902,7 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32 {const unsigned char *index_fanout = p->index_data;const unsigned char *index_lookup; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;int index_lookup_width;if (!index_fanout)
@@ -1922,7 +1927,7 @@ int nth_packed_object_id(struct object_id *oid, uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (!index) {if (open_pack_index (p))return -1 ;
@@ -1933,11 +1938,10 @@ int nth_packed_object_id(struct object_id *oid, index +=4 *256 ;if (p->index_version ==1 ) {oidread (oid, index +st_add (st_mult (hashsz +4 , n),4 ), - the_repository->hash_algo); + p->repo->hash_algo); }else { index +=8 ; -oidread (oid, index +st_mult (hashsz, n), - the_repository->hash_algo); +oidread (oid, index +st_mult (hashsz, n), p->repo->hash_algo); }return 0 ; }
@@ -1959,7 +1963,7 @@ void check_pack_index_ptr(const struct packed_git *p, const void *vptr)off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz; index +=4 *256 ;if (p->index_version ==1 ) {return ntohl (*((uint32_t *)(index +st_mult (hashsz +4 , n))));
@@ -2159,7 +2163,7 @@ int for_each_object_in_pack(struct packed_git *p,int r =0 ;if (flags &FOR_EACH_OBJECT_PACK_ORDER ) { -if (load_pack_revindex (the_repository, p)) +if (load_pack_revindex (p->repo, p))return -1 ; }
@@ -2227,7 +2231,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, }static int add_promisor_object (const struct object_id *oid, - struct packed_git *pack UNUSED, +struct packed_git *pack, uint32_t pos UNUSED,void *set_) {
@@ -2235,12 +2239,12 @@ static int add_promisor_object(const struct object_id *oid,struct object *obj;int we_parsed_object; - obj =lookup_object (the_repository, oid); + obj =lookup_object (pack->repo, oid);if (obj && obj->parsed) { we_parsed_object =0 ; }else { we_parsed_object =1 ; - obj =parse_object (the_repository, oid); + obj =parse_object (pack->repo, oid); }if (!obj)
Signature · 1 line
2.47.0
PATCH v7 3/9 packfile: pass `repository` to static function in the file
In reply to Karthik NayakSome of the static functions in the packfile.c access global variables, which can simply be avoiding by passing the repository struct down to them. Let's do that.
--- packfile.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
packfile.c +8 -7
@@ -460,13 +460,13 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc *accept_windows_inuse = has_windows_inuse; } -static int close_one_pack (void ) +static int close_one_pack (struct repository *r) {struct packed_git *p, *lru_p =NULL ;struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = the_repository->objects->packed_git; p; p = p->next) { +for (p = r->objects->packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -555,7 +555,7 @@ static int open_packed_git_1(struct packed_git *p) pack_max_fds =1 ; } -while (pack_max_fds <= pack_open_fds &&close_one_pack ()) +while (pack_max_fds <= pack_open_fds &&close_one_pack (p->repo)) ;/* nothing */ p->pack_fd =git_open (p->pack_name);
@@ -610,7 +610,8 @@ static int open_packed_git(struct packed_git *p)return -1 ; } -static int in_window (struct pack_window *win,off_t offset) +static int in_window (struct repository *r,struct pack_window *win, +off_t offset) {/* We must promise at least one full hash after the * offset is available from this window, otherwise the offset
@@ -620,7 +621,7 @@ static int in_window(struct pack_window *win, off_t offset)*/ off_t win_off = win->offset;return win_off <= offset - && (offset + the_hash_algo->rawsz) <= (win_off + win->len); + && (offset + r->hash_algo->rawsz) <= (win_off + win->len); } unsigned char *use_pack(struct packed_git *p,
@@ -642,11 +643,11 @@ unsigned char *use_pack(struct packed_git *p,if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" )); -if (!win || !in_window (win, offset)) { +if (!win || !in_window (p->repo, win, offset)) {if (win) win->inuse_cnt--;for (win = p->windows; win; win = win->next) { -if (in_window (win, offset)) +if (in_window (p->repo, win, offset))break ; }if (!win) {
Signature · 1 line
2.47.0
PATCH v7 4/9 packfile: pass down repository to `odb_pack_name`
In reply to Karthik NayakThe function odb_pack_name currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers.
--- builtin/fast-import.c | 8 ++++---- builtin/index-pack.c | 4 ++-- builtin/pack-redundant.c | 2 +- http.c | 2 +- packfile.c | 9 ++++----- packfile.h | 3 ++- 6 files changed, 14 insertions(+), 14 deletions(-)
builtin/fast-import.c +4 -4
@@ -806,7 +806,7 @@ static char *keep_pack(const char *curr_index_name)struct strbuf name =STRBUF_INIT ;int keep_fd; -odb_pack_name (&name, pack_data->hash,"keep" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"keep" ); keep_fd =odb_pack_keep (name.buf);if (keep_fd <0 )die_errno ("cannot create keep file" );
@@ -814,11 +814,11 @@ static char *keep_pack(const char *curr_index_name)if (close (keep_fd))die_errno ("failed to write keep file" ); -odb_pack_name (&name, pack_data->hash,"pack" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"pack" );if (finalize_object_file (pack_data->pack_name, name.buf))die ("cannot store pack file" ); -odb_pack_name (&name, pack_data->hash,"idx" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"idx" );if (finalize_object_file (curr_index_name, name.buf))die ("cannot store index file" );free ((void *)curr_index_name);
@@ -832,7 +832,7 @@ static void unkeep_all_packs(void)for (k =0 ; k < pack_id; k++) {struct packed_git *p = all_packs[k]; -odb_pack_name (&name, p->hash,"keep" ); +odb_pack_name (p->repo, &name, p->hash,"keep" );unlink_or_warn (name.buf); }strbuf_release (&name);
builtin/index-pack.c +2 -2
@@ -1479,7 +1479,7 @@ static void write_special_file(const char *suffix, const char *msg,if (pack_name) filename =derive_filename (pack_name,"pack" , suffix, &name_buf);else - filename =odb_pack_name (&name_buf, hash, suffix); + filename =odb_pack_name (the_repository, &name_buf, hash, suffix); fd =odb_pack_keep (filename);if (fd <0 ) {
@@ -1507,7 +1507,7 @@ static void rename_tmp_packfile(const char **final_name, {if (!*final_name ||strcmp (*final_name, curr_name)) {if (!*final_name) - *final_name =odb_pack_name (name, hash, ext); + *final_name =odb_pack_name (the_repository, name, hash, ext);if (finalize_object_file (curr_name, *final_name))die (_ ("unable to rename temporary '*.%s' file to '%s'" ), ext, *final_name);
builtin/pack-redundant.c +1 -1
@@ -690,7 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix UNUSED, s pl = red =pack_list_difference (local_packs, min);while (pl) {printf ("%s\n%s\n" , -odb_pack_name (&idx_name, pl->pack->hash,"idx" ), +odb_pack_name (pl->pack->repo, &idx_name, pl->pack->hash,"idx" ), pl->pack->pack_name); pl = pl->next; }
http.c +1 -1
@@ -2581,7 +2581,7 @@ struct http_pack_request *new_direct_http_pack_request( preq->url = url; -odb_pack_name (&preq->tmpfile, packed_git_hash,"pack" ); +odb_pack_name (the_repository, &preq->tmpfile, packed_git_hash,"pack" );strbuf_addstr (&preq->tmpfile,".temp" ); preq->packfile =fopen (preq->tmpfile.buf,"a" );if (!preq->packfile) {
packfile.c +4 -5
@@ -25,13 +25,12 @@#include "pack-revindex.h" #include "promisor-remote.h" -char *odb_pack_name (struct strbuf *buf, -const unsigned char *hash, -const char *ext) +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext) {strbuf_reset (buf); -strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (the_repository), -hash_to_hex (hash), ext); +strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (r), +hash_to_hex_algop (hash, r->hash_algo), ext);return buf->buf; }
packfile.h +2 -1
@@ -29,7 +29,8 @@ struct pack_entry {* * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" */ -char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext);/* * Return the basename of the packfile, omitting any containing directory
Signature · 1 line
2.47.0
PATCH v7 5/9 packfile: pass down repository to `has_object[_kept]_pack`
In reply to Karthik NayakThe functions has_object[_kept]_pack currently rely on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from these functions and any related ones.
--- builtin/count-objects.c | 2 +- builtin/fsck.c | 2 +- builtin/pack-objects.c | 4 ++-- diff.c | 3 ++- list-objects.c | 3 ++- pack-bitmap.c | 2 +- packfile.c | 9 +++++---- packfile.h | 5 +++-- prune-packed.c | 2 +- reachable.c | 2 +- revision.c | 4 ++-- 11 files changed, 21 insertions(+), 17 deletions(-)
builtin/count-objects.c +1 -1
@@ -67,7 +67,7 @@ static int count_loose(const struct object_id *oid, const char *path,else { loose_size +=on_disk_bytes (st); loose++; -if (verbose &&has_object_pack (oid)) +if (verbose &&has_object_pack (the_repository, oid)) packed_loose++; }return 0 ;
builtin/fsck.c +1 -1
@@ -272,7 +272,7 @@ static void check_reachable_object(struct object *obj)if (!(obj->flags &HAS_OBJ )) {if (is_promisor_object (&obj->oid))return ; -if (has_object_pack (&obj->oid)) +if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */ printf_ln(_("missing %s %s"), printable_type(&obj->oid, obj->type),
builtin/pack-objects.c +2 -2
@@ -1529,7 +1529,7 @@ static int want_found_object(const struct object_id *oid, int exclude,return 0 ;if (ignore_packed_keep_in_core && p->pack_keep_in_core)return 0 ; -if (has_object_kept_pack (oid, flags)) +if (has_object_kept_pack (p->repo, oid, flags))return 0 ; }
@@ -3627,7 +3627,7 @@ static void show_cruft_commit(struct commit *commit, void *data) static int cruft_include_check_obj(struct object *obj, void *data UNUSED) { -return !has_object_kept_pack (&obj->oid,IN_CORE_KEEP_PACKS ); +return !has_object_kept_pack (to_pack.repo, &obj->oid,IN_CORE_KEEP_PACKS ); }static int cruft_include_check (struct commit *commit,void *data)
diff.c +2 -1
@@ -4041,7 +4041,8 @@ static int reuse_worktree_file(struct index_state *istate,* objects however would tend to be slower as they need * to be individually opened and inflated. */ -if (!FAST_WORKING_DIRECTORY && !want_file &&has_object_pack (oid)) +if (!FAST_WORKING_DIRECTORY && !want_file && +has_object_pack (istate->repo, oid))return 0 ;/*
list-objects.c +2 -1
@@ -41,7 +41,8 @@ static void show_object(struct traversal_context *ctx, {if (!ctx->show_object)return ; -if (ctx->revs->unpacked &&has_object_pack (&object->oid)) +if (ctx->revs->unpacked &&has_object_pack (ctx->revs->repo, + &object->oid))return ; ctx->show_object (object, name, ctx->show_data);
pack-bitmap.c +1 -1
@@ -1889,7 +1889,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,bitmap_unset (result, i);for (i =0 ; i < eindex->count; ++i) { -if (has_object_pack (&eindex->objects[i]->oid)) +if (has_object_pack (the_repository, &eindex->objects[i]->oid))bitmap_unset (result, objects_nr + i); } }
packfile.c +5 -4
@@ -2143,16 +2143,17 @@ int find_kept_pack_entry(struct repository *r,return 0 ; } -int has_object_pack (const struct object_id *oid) +int has_object_pack (struct repository *r,const struct object_id *oid) {struct pack_entry e; -return find_pack_entry (the_repository, oid, &e); +return find_pack_entry (r, oid, &e); } -int has_object_kept_pack (const struct object_id *oid,unsigned flags) +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags) {struct pack_entry e; -return find_kept_pack_entry (the_repository, oid, flags, &e); +return find_kept_pack_entry (r, oid, flags, &e); } int for_each_object_in_pack(struct packed_git *p,
packfile.h +3 -2
@@ -193,8 +193,9 @@ const struct packed_git *has_packed_and_bad(struct repository *, const struct obint find_pack_entry (struct repository *r,const struct object_id *oid,struct pack_entry *e);int find_kept_pack_entry (struct repository *r,const struct object_id *oid,unsigned flags,struct pack_entry *e); -int has_object_pack (const struct object_id *oid); -int has_object_kept_pack (const struct object_id *oid,unsigned flags); +int has_object_pack (struct repository *r,const struct object_id *oid); +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags);/* * Return 1 if an object in a promisor packfile is or refers to the given
prune-packed.c +1 -1
@@ -24,7 +24,7 @@ static int prune_object(const struct object_id *oid, const char *path, {int *opts = data; -if (!has_object_pack (oid)) +if (!has_object_pack (the_repository, oid))return 0 ;if (*opts &PRUNE_PACKED_DRY_RUN )
reachable.c +1 -1
@@ -239,7 +239,7 @@ static int want_recent_object(struct recent_data *data, const struct object_id *oid) {if (data->ignore_in_core_kept_packs && -has_object_kept_pack (oid,IN_CORE_KEEP_PACKS )) +has_object_kept_pack (data->revs->repo, oid,IN_CORE_KEEP_PACKS ))return 0 ;return 1 ; }
revision.c +2 -2
@@ -4103,10 +4103,10 @@ enum commit_action get_commit_action(struct rev_info *revs, struct commit *commi {if (commit->object.flags &SHOWN )return commit_ignore; -if (revs->unpacked &&has_object_pack (&commit->object.oid)) +if (revs->unpacked &&has_object_pack (revs->repo, &commit->object.oid))return commit_ignore;if (revs->no_kept_objects) { -if (has_object_kept_pack (&commit->object.oid, +if (has_object_kept_pack (revs->repo, &commit->object.oid, revs->keep_pack_cache_flags))return commit_ignore; }
Signature · 1 line
2.47.0
PATCH v7 6/9 packfile: pass down repository to `for_each_packed_object`
In reply to Karthik NayakThe function for_each_packed_object currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from this function and closely related function is_promisor_object.
--- builtin/cat-file.c | 7 ++++--- builtin/fsck.c | 18 +++++++++++------- builtin/pack-objects.c | 7 +++++-- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 2 +- fsck.c | 2 +- list-objects.c | 4 ++-- object-store-ll.h | 4 ++-- packfile.c | 14 +++++++------- packfile.h | 2 +- promisor-remote.c | 2 +- reachable.c | 2 +- revision.c | 9 +++++---- tag.c | 2 +- 15 files changed, 44 insertions(+), 35 deletions(-)
builtin/cat-file.c +4 -3
@@ -827,15 +827,16 @@ static int batch_objects(struct batch_options *opt) cb.seen = &seen;for_each_loose_object (batch_unordered_loose, &cb,0 ); -for_each_packed_object (batch_unordered_packed, &cb, -FOR_EACH_OBJECT_PACK_ORDER ); +for_each_packed_object (the_repository, batch_unordered_packed, + &cb,FOR_EACH_OBJECT_PACK_ORDER );oidset_clear (&seen); }else {struct oid_array sa =OID_ARRAY_INIT ;for_each_loose_object (collect_loose_object, &sa,0 ); -for_each_packed_object (collect_packed_object, &sa,0 ); +for_each_packed_object (the_repository, collect_packed_object, + &sa,0 );oid_array_for_each_unique (&sa, batch_object_cb, &cb);
builtin/fsck.c +11 -7
@@ -150,7 +150,7 @@ static int mark_object(struct object *obj, enum object_type type,return 0 ; obj->flags |=REACHABLE ; -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))/* * Further recursion does not need to be performed on this * object since it is a promisor object (so it does not need to
@@ -270,7 +270,7 @@ static void check_reachable_object(struct object *obj)* do a full fsck */ if (!(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */
@@ -391,7 +391,10 @@ static void check_connectivity(void)* traversal. */ for_each_loose_object (mark_loose_unreachable_referents,NULL ,0 ); -for_each_packed_object (mark_packed_unreachable_referents,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_unreachable_referents, +NULL , +0 ); }/* Look up all the requirements, warn about missing objects.. */
@@ -488,7 +491,7 @@ static void fsck_handle_reflog_oid(const char *refname, struct object_id *oid, refname, timestamp); obj->flags |=USED ;mark_object_reachable (obj); - }else if (!is_promisor_object (oid)) { + }else if (!is_promisor_object (the_repository, oid)) {error (_ ("%s: invalid reflog entry %s" ), refname,oid_to_hex (oid)); errors_found |=ERROR_REACHABLE ;
@@ -531,7 +534,7 @@ static int fsck_handle_ref(const char *refname, const char *referent UNUSED, con obj =parse_object (the_repository, oid);if (!obj) { -if (is_promisor_object (oid)) { +if (is_promisor_object (the_repository, oid)) {/* * Increment default_refs anyway, because this is a * valid ref.
@@ -966,7 +969,8 @@ int cmd_fsck(int argc,if (connectivity_only) {for_each_loose_object (mark_loose_for_connectivity,NULL ,0 ); -for_each_packed_object (mark_packed_for_connectivity,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_for_connectivity,NULL ,0 ); }else {prepare_alt_odb (the_repository);for (odb = the_repository->objects->odb; odb; odb = odb->next)
@@ -1011,7 +1015,7 @@ int cmd_fsck(int argc, &oid);if (!obj || !(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&oid)) +if (is_promisor_object (the_repository, &oid))continue ;error (_ ("%s: object missing" ),oid_to_hex (&oid)); errors_found |=ERROR_OBJECT ;
builtin/pack-objects.c +5 -2
@@ -3858,7 +3858,8 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,* Quietly ignore EXPECTED missing objects. This avoids problems with * staging them now and getting an odd error later. */ -if (!has_object (the_repository, &obj->oid,0 ) &&is_promisor_object (&obj->oid)) +if (!has_object (the_repository, &obj->oid,0 ) && +is_promisor_object (to_pack.repo, &obj->oid))return ;show_object (obj, name, data);
@@ -3927,7 +3928,9 @@ static int add_object_in_unpacked_pack(const struct object_id *oid,static void add_objects_in_unpacked_packs (void ) { - if (for_each_packed_object(add_object_in_unpacked_pack, NULL, + if (for_each_packed_object(to_pack.repo, + add_object_in_unpacked_pack, + NULL, FOR_EACH_OBJECT_PACK_ORDER | FOR_EACH_OBJECT_LOCAL_ONLY | FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS |
builtin/repack.c +1 -1
@@ -404,7 +404,7 @@ static void repack_promisor_objects(const struct pack_objects_args *args,* {type -> existing pack order} ordering when computing deltas instead * of a {type -> size} ordering, which may produce better deltas. */ -for_each_packed_object (write_oid, &cmd, +for_each_packed_object (the_repository, write_oid, &cmd,FOR_EACH_OBJECT_PROMISOR_ONLY );if (cmd.in ==-1 ) {
builtin/rev-list.c +1 -1
@@ -121,7 +121,7 @@ static inline void finish_object__ma(struct object *obj)return ;case MA_ALLOW_PROMISOR : -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;die ("unexpected missing %s object '%s'" ,type_name (obj->type),oid_to_hex (&obj->oid));
commit-graph.c +1 -1
@@ -1960,7 +1960,7 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx) ctx->progress =start_delayed_progress (_ ("Finding commits for commit graph among packed objects" ), ctx->approx_nr_objects); -for_each_packed_object (add_packed_commits, ctx, +for_each_packed_object (ctx->r, add_packed_commits, ctx,FOR_EACH_OBJECT_PACK_ORDER );if (ctx->progress_done < ctx->approx_nr_objects)display_progress (ctx->progress, ctx->approx_nr_objects);
fsck.c +1 -1
@@ -1295,7 +1295,7 @@ static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done, buf =repo_read_object_file (the_repository, oid, &type, &size);if (!buf) { -if (is_promisor_object (oid)) +if (is_promisor_object (the_repository, oid))continue ; ret |= report(options, oid, OBJ_BLOB, msg_missing,
list-objects.c +2 -2
@@ -75,7 +75,7 @@ static void process_blob(struct traversal_context *ctx,*/ if (ctx->revs->exclude_promisor_objects && !repo_has_object_file (the_repository, &obj->oid) && -is_promisor_object (&obj->oid)) +is_promisor_object (ctx->revs->repo, &obj->oid))return ; pathlen = path->len;
@@ -180,7 +180,7 @@ static void process_tree(struct traversal_context *ctx,* an incomplete list of missing objects. */ if (revs->exclude_promisor_objects && -is_promisor_object (&obj->oid)) +is_promisor_object (revs->repo, &obj->oid))return ;if (!revs->do_not_die_on_missing_objects)
object-store-ll.h +2 -2
@@ -550,7 +550,7 @@ typedef int each_packed_object_fn(const struct object_id *oid,int for_each_object_in_pack (struct packed_git *p,each_packed_object_fn ,void *data,enum for_each_object_flags flags); -int for_each_packed_object (each_packed_object_fn ,void *, -enum for_each_object_flags flags); +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags);#endif /* OBJECT_STORE_LL_H */
packfile.c +7 -7
@@ -2200,15 +2200,15 @@ int for_each_object_in_pack(struct packed_git *p,return r; } -int for_each_packed_object (each_packed_object_fn cb,void *data, -enum for_each_object_flags flags) +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags) {struct packed_git *p;int r =0 ;int pack_errors =0 ; -prepare_packed_git (the_repository); -for (p =get_all_packs (the_repository); p; p = p->next) { +prepare_packed_git (repo); +for (p =get_all_packs (repo); p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ; if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
@@ -2286,14 +2286,14 @@ static int add_promisor_object(const struct object_id *oid,return 0 ; } -int is_promisor_object (const struct object_id *oid) +int is_promisor_object (struct repository *r,const struct object_id *oid) {static struct oidset promisor_objects;static int promisor_objects_prepared;if (!promisor_objects_prepared) { -if (repo_has_promisor_remote (the_repository)) { -for_each_packed_object (add_promisor_object, +if (repo_has_promisor_remote (r)) { +for_each_packed_object (r, add_promisor_object, &promisor_objects,FOR_EACH_OBJECT_PROMISOR_ONLY |FOR_EACH_OBJECT_PACK_ORDER );
packfile.h +1 -1
@@ -201,7 +201,7 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,* Return 1 if an object in a promisor packfile is or refers to the given * object, 0 otherwise. */ -int is_promisor_object (const struct object_id *oid); +int is_promisor_object (struct repository *r,const struct object_id *oid);/* * Expose a function for fuzz testing.
promisor-remote.c +1 -1
@@ -283,7 +283,7 @@ void promisor_remote_get_direct(struct repository *repo, }for (i =0 ; i < remaining_nr; i++) { -if (is_promisor_object (&remaining_oids[i])) +if (is_promisor_object (repo, &remaining_oids[i]))die (_ ("could not fetch %s from promisor remote" ),oid_to_hex (&remaining_oids[i])); }
reachable.c +1 -1
@@ -324,7 +324,7 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,if (ignore_in_core_kept_packs) flags |=FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS ; - r =for_each_packed_object (add_recent_packed, &data, flags); + r =for_each_packed_object (revs->repo, add_recent_packed, &data, flags); done:oidset_clear (&data.extra_recent_oids);
revision.c +5 -4
@@ -390,7 +390,8 @@ static struct object *get_reference(struct rev_info *revs, const char *name,if (!object) {if (revs->ignore_missing)return NULL ; -if (revs->exclude_promisor_objects &&is_promisor_object (oid)) +if (revs->exclude_promisor_objects && +is_promisor_object (revs->repo, oid))return NULL ;if (revs->do_not_die_on_missing_objects) {oidset_insert (&revs->missing_commits, oid);
@@ -432,7 +433,7 @@ static struct commit *handle_commit(struct rev_info *revs,if (revs->ignore_missing_links || (flags &UNINTERESTING ))return NULL ;if (revs->exclude_promisor_objects && -is_promisor_object (&tag->tagged->oid)) +is_promisor_object (revs->repo, &tag->tagged->oid))return NULL ;if (revs->do_not_die_on_missing_objects && oid) {oidset_insert (&revs->missing_commits, oid);
@@ -1211,7 +1212,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit, revs->do_not_die_on_missing_objects;if (repo_parse_commit_gently (revs->repo, p, gently) <0 ) {if (revs->exclude_promisor_objects && -is_promisor_object (&p->object.oid)) { +is_promisor_object (revs->repo, &p->object.oid)) {if (revs->first_parent_only)break ;continue ;
@@ -3915,7 +3916,7 @@ int prepare_revision_walk(struct rev_info *revs) revs->treesame.name ="treesame" ;if (revs->exclude_promisor_objects) { -for_each_packed_object (mark_uninteresting, revs, +for_each_packed_object (revs->repo, mark_uninteresting, revs,FOR_EACH_OBJECT_PROMISOR_ONLY ); }
tag.c +1 -1
@@ -84,7 +84,7 @@ struct object *deref_tag(struct repository *r, struct object *o, const char *war o =NULL ; }if (!o && warn) { -if (last_oid &&is_promisor_object (last_oid)) +if (last_oid &&is_promisor_object (r, last_oid))return NULL ;if (!warnlen) warnlen =strlen (warn);
Signature · 1 line
2.47.0
Re: [PATCH v7 6/9] packfile: pass down repository to `for_each_packed_object`
Quoted 14 lines · Karthik Nayak
packfile.c +2
-2@@ -2200,15 +2200,15 @@ int for_each_object_in_pack(struct packed_git *p,return r; } -int for_each_packed_object (each_packed_object_fn cb,void *data, -enum for_each_object_flags flags) +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags) {struct packed_git *p;
packfile.c int for_each_object_in_pack(struct packed_git *p, · line 2214
Quoted 10 lines
packfile.c +2
-2int r =0 ;int pack_errors =0 ; -prepare_packed_git (the_repository); -for (p =get_all_packs (the_repository); p; p = p->next) { +prepare_packed_git (repo); +for (p =get_all_packs (repo); p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ; if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
Not the fault of your series, but this prepare_packed_git() call is unnecessary, since it is the first thing that get_all_packs() does when it executes.
I suspect that this call comes from way back in 660c889e46 (sha1_file:
add for_each iterators for loose and packed objects, 2014-10-15). It could have been removed in 454ea2e4d7 (treewide: use get_all_packs, 2018-08-20), but I think that patch was a straightforward conversion that did not inspect each individual change.
Anyway, nothing to do immediately here, but just something I saw when reviewing and figured was worth writing down somewhere.
Thanks,
Taylor
PATCH 1/1 packfile.c: remove unnecessary prepare_packed_git() call
In 454ea2e4d7 (treewide: use get_all_packs, 2018-08-20) we converted existing calls to both:
- get_packed_git(), as well as
- the_repository->objects->packed_git
, to instead use the new get_all_packs() function.
In the instance that this commit addresses, there was a preceding call to prepare_packed_git(), which dates all the way back to 660c889e46 (sha1_file: add for_each iterators for loose and packed objects, 2014-10-15) when its caller (for_each_packed_object()) was first introduced.
This call could have been removed in 454ea2e4d7, since get_all_packs() itself calls prepare_packed_git(). But the translation in 454ea2e4d7 was (to the best of my knowledge) a find-and-replace rather than inspecting each individual caller.
Having an extra prepare_packed_git() call here is harmless, since it will notice that we have already set the 'packed_git_initialized' field and the call will be a noop. So we're only talking about a few dozen CPU cycles to set up and tear down the stack frame.
But having a lone prepare_packed_git() call immediately before a call to get_all_packs() confused me, so let's remove it as redundant to avoid more confusion in the future.
--- packfile.c | 1 - 1 file changed, 1 deletion(-)
packfile.c +0 -1
@@ -2223,7 +2223,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,int r =0 ;int pack_errors =0 ; -prepare_packed_git (repo);for (p =get_all_packs (repo); p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ;
base-commit: 78e4bc3efc49ee4c9ec1ce14117c2e7d99999234 --
This applies on top of Karthik's series, and was something small I noticed while reading it.
2.47.0.237.gc601277f4c4
Re: [PATCH] packfile.c: remove unnecessary prepare_packed_git() call
Quoted 10 lines · Taylor Blau
In the instance that this commit addresses, there was a preceding call to
prepare_packed_git(), which dates all the way back to660c889e46(sha1_file: addfor_eachiterators for loose and packed objects, 2014-10-15) when its caller (for_each_packed_object()) was first introduced.This call could have been removed in
454ea2e4d7, sinceget_all_packs()itself callsprepare_packed_git(). But the translation in454ea2e4d7was (to the best of my knowledge) a find-and-replace rather than inspecting each individual caller.
Yeah, I think that describes what happened.
Quoted 8 lines
Having an extra
prepare_packed_git()call here is harmless, since it will notice that we have already set the 'packed_git_initialized' field and the call will be a noop. So we're only talking about a few dozen CPU cycles to set up and tear down the stack frame.But having a lone
prepare_packed_git()call immediately before a call toget_all_packs()confused me, so let's remove it as redundant to avoid more confusion in the future.
Agreed. I think this is worth doing.
I briefly grepped for other cases. This one confused me:
builtin/gc.c=1272=static off_t get_auto_pack_size(void)
--
builtin/gc.c-1292- reprepare_packed_git(r);
builtin/gc.c:1293: for (p = get_all_packs(r); p; p = p->next) {It's not a noop because it's calling the reprepare() function, which will re-check the directory. But why? Are we expecting that something changed? This is called only when making the midx, so maybe it's trying to refresh the set of packs after repacking. But that seems like something that should happen explicitly, not as a side effect of an otherwise read-only function.
Removing it still passes the tests. So I dunno. It looks superfluous to me, but it's perhaps more risky than the one you identified.
-Peff
PATCH v7 7/9 config: make `delta_base_cache_limit` a non-global variable
In reply to Karthik NayakThe delta_base_cache_limit variable is a global config variable used by multiple subsystems. Let's make this non-global, by adding this variable to the stack of each of the subsystems where it is used.
In gc.c we add it to the gc_config struct and also the constructor function. In index-pack.c we add it to the pack_idx_option struct and its constructor. Finally, in packfile.c we dynamically retrieve this value from the repository config, since the value is only used once in the entire subsystem.
These changes are made to remove the usage of delta_base_cache_limit as a global variable in packfile.c. This brings us one step closer to removing the USE_THE_REPOSITORY_VARIABLE definition in packfile.c which we complete in the next patch.
--- builtin/gc.c | 8 +++++++- builtin/index-pack.c | 10 +++++++--- config.c | 5 ----- environment.c | 1 - environment.h | 1 - pack-objects.h | 3 ++- pack-write.c | 1 + pack.h | 1 + packfile.c | 13 +++++++++++-- 9 files changed, 29 insertions(+), 14 deletions(-)
builtin/gc.c +7 -1
@@ -138,6 +138,7 @@ struct gc_config {char *repack_filter_to;unsigned long big_pack_threshold;unsigned long max_delta_cache_size; +size_t delta_base_cache_limit; };#define GC_CONFIG_INIT { \
@@ -153,6 +154,7 @@ struct gc_config { .prune_expire = xstrdup("2.weeks.ago"), \ .prune_worktrees_expire = xstrdup("3.months.ago"), \ .max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \ + .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \ }static void gc_config_release (struct gc_config *cfg)
@@ -168,6 +170,7 @@ static void gc_config(struct gc_config *cfg) {const char *value;char *owned =NULL ; +unsigned long ulongval;if (!git_config_get_value ("gc.packrefs" , &value)) {if (value && !strcmp (value,"notbare" ))
@@ -206,6 +209,9 @@ static void gc_config(struct gc_config *cfg)git_config_get_ulong ("gc.bigpackthreshold" , &cfg->big_pack_threshold);git_config_get_ulong ("pack.deltacachesize" , &cfg->max_delta_cache_size); +if (!git_config_get_ulong ("core.deltabasecachelimit" , &ulongval)) + cfg->delta_base_cache_limit = ulongval; +if (!git_config_get_string ("gc.repackfilter" , &owned)) {free (cfg->repack_filter); cfg->repack_filter = owned;
@@ -416,7 +422,7 @@ static uint64_t estimate_repack_memory(struct gc_config *cfg,* read_sha1_file() (either at delta calculation phase, or * writing phase) also fills up the delta base cache */ - heap += delta_base_cache_limit; + heap += cfg->delta_base_cache_limit;/* and of course pack-objects has its own delta cache */ heap += cfg->max_delta_cache_size;
builtin/index-pack.c +7 -3
@@ -1238,7 +1238,7 @@ static void parse_pack_objects(unsigned char *hash)* recursively checking if the resulting object is used as a base * for some more deltas. */ -static void resolve_deltas (void ) +static void resolve_deltas (struct pack_idx_option *opts) {int i;
@@ -1254,7 +1254,7 @@ static void resolve_deltas(void) nr_ref_deltas + nr_ofs_deltas); nr_dispatched =0 ; - base_cache_limit = delta_base_cache_limit * nr_threads; + base_cache_limit = opts->delta_base_cache_limit * nr_threads;if (nr_threads >1 ||getenv ("GIT_FORCE_THREADS" )) {init_thread ();work_lock ();
@@ -1604,6 +1604,10 @@ static int git_index_pack_config(const char *k, const char *v,else opts->flags &= ~WRITE_REV ; } +if (!strcmp (k,"core.deltabasecachelimit" )) { + opts->delta_base_cache_limit =git_config_ulong (k, v, ctx->kvi); +return 0 ; + }return git_default_config (k, v, ctx, cb); }
@@ -1930,7 +1934,7 @@ int cmd_index_pack(int argc,parse_pack_objects (pack_hash);if (report_end_of_input)write_in_full (2 ,"\0" ,1 ); -resolve_deltas (); +resolve_deltas (&opts);conclude_pack (fix_thin_pack, curr_pack, pack_hash);free (ofs_deltas);free (ref_deltas);
config.c +0 -5
@@ -1515,11 +1515,6 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.deltabasecachelimit" )) { - delta_base_cache_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -1
@@ -51,7 +51,6 @@ enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ;size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ;size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ; -size_t delta_base_cache_limit =96 *1024 *1024 ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
environment.h +0 -1
@@ -165,7 +165,6 @@ extern int zlib_compression_level;extern int pack_compression_level;extern size_t packed_git_window_size;extern size_t packed_git_limit; -extern size_t delta_base_cache_limit;extern unsigned long big_file_threshold;extern unsigned long pack_size_limit_cfg;extern int max_allowed_tree_depth;
pack-objects.h +2 -1
@@ -7,7 +7,8 @@struct repository ; -#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_BASE_CACHE_LIMIT (96 * 1024 * 1024)#define OE_DFS_STATE_BITS 2#define OE_DEPTH_BITS 12
pack-write.c +1 -0
@@ -21,6 +21,7 @@ void reset_pack_idx_option(struct pack_idx_option *opts)memset (opts,0 ,sizeof (*opts)); opts->version =2 ; opts->off32_limit =0x7fffffff ; + opts->delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; }static int sha1_compare (const void *_a,const void *_b)
pack.h +1 -0
@@ -58,6 +58,7 @@ struct pack_idx_option {*/ int anomaly_alloc, anomaly_nr;uint32_t *anomaly; +unsigned long delta_base_cache_limit; };void reset_pack_idx_option (struct pack_idx_option *);
packfile.c +11 -2
@@ -24,6 +24,8 @@#include "commit-graph.h" #include "pack-revindex.h" #include "promisor-remote.h" +#include "config.h" +#include "pack-objects.h" char *odb_pack_name (struct repository *r,struct strbuf *buf,const unsigned char *hash,const char *ext)
@@ -1496,7 +1498,9 @@ void clear_delta_base_cache(void) }static void add_delta_base_cache (struct packed_git *p,off_t base_offset, -void *base,unsigned long base_size,enum object_type type) +void *base,unsigned long base_size, +unsigned long delta_base_cache_limit, +enum object_type type) {struct delta_base_cache_entry *ent;struct list_head *lru, *tmp;
@@ -1697,6 +1701,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,struct unpack_entry_stack_ent *delta_stack = small_delta_stack;int delta_stack_nr =0 , delta_stack_alloc =UNPACK_ENTRY_STACK_PREALLOC ;int base_from_cache =0 ; +unsigned long delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; + +repo_config_get_ulong (r,"core.deltabasecachelimit" , &delta_base_cache_limit);write_pack_access_log (p, obj_offset);
@@ -1878,7 +1885,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,* before we are done using it. */ if (!external_base) -add_delta_base_cache (p, base_obj_offset, base, base_size, type); +add_delta_base_cache (p, base_obj_offset, base, + base_size, delta_base_cache_limit, + type);free (delta_data);free (external_base);
Signature · 1 line
2.47.0
Re: [PATCH v7 7/9] config: make `delta_base_cache_limit` a non-global variable
packfile.c void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset, · line 1709
Quoted 10 lines · Karthik Nayak
packfile.c +3
-0@@ -1697,6 +1701,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,struct unpack_entry_stack_ent *delta_stack = small_delta_stack;int delta_stack_nr =0 , delta_stack_alloc =UNPACK_ENTRY_STACK_PREALLOC ;int base_from_cache =0 ; +unsigned long delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; + +repo_config_get_ulong (r,"core.deltabasecachelimit" , &delta_base_cache_limit);write_pack_access_log (p, obj_offset);
Hmm. This repo_config_get_ulong() call will look for the configset entry in a hashmap which is faster than parsing the configuration file from scratch every time, but still expensive for my taste in a function as hot as unpack_entry().
Should this also go in the_repository->settings instead? That way we have a single field access instead of a hashmap lookup (with multiple layers of function calls between us and the actual lookup).
Thanks,
Taylor
Re: [PATCH v7 7/9] config: make `delta_base_cache_limit` a non-global variable
Quoted 17 lines · Taylor Blau
packfile.c void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset, · line 1709
Quoted 10 lines · Karthik Nayak
packfile.c +3
-0@@ -1697,6 +1701,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,struct unpack_entry_stack_ent *delta_stack = small_delta_stack;int delta_stack_nr =0 , delta_stack_alloc =UNPACK_ENTRY_STACK_PREALLOC ;int base_from_cache =0 ; +unsigned long delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; + +repo_config_get_ulong (r,"core.deltabasecachelimit" , &delta_base_cache_limit);write_pack_access_log (p, obj_offset);Hmm. This
repo_config_get_ulong()call will look for the configset entry in a hashmap which is faster than parsing the configuration file from scratch every time, but still expensive for my taste in a function as hot asunpack_entry().
Quoted 3 lines
Should this also go in
the_repository->settingsinstead? That way we have a single field access instead of a hashmap lookup (with multiple layers of function calls between us and the actual lookup).
Good catch. I missed this one when making a similar argument for the packed_git variables.
-Peff
Re: [PATCH v7 7/9] config: make `delta_base_cache_limit` a non-global variable
In reply to Taylor BlauQuoted 9 lines · Taylor Blau
Quoted 8 lines · Karthik Nayak
packfile.c +3
-0@@ -1697,6 +1701,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,struct unpack_entry_stack_ent *delta_stack = small_delta_stack;int delta_stack_nr =0 , delta_stack_alloc =UNPACK_ENTRY_STACK_PREALLOC ;int base_from_cache =0 ; +unsigned long delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; + +repo_config_get_ulong (r,"core.deltabasecachelimit" , &delta_base_cache_limit);
Quoted 8 lines
packfile.c void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset, · line 1709
Quoted 2 lines
packfile.c +0
-0write_pack_access_log (p, obj_offset);Hmm. This
repo_config_get_ulong()call will look for the configset entry in a hashmap which is faster than parsing the configuration file from scratch every time, but still expensive for my taste in a function as hot asunpack_entry().
Thanks for pointing out, I do have not much idea about the object database code, so any input here is appreciated.
Quoted 4 lines
Should this also go in
the_repository->settingsinstead? That way we have a single field access instead of a hashmap lookup (with multiple layers of function calls between us and the actual lookup).
I think that is the best way, though we'll have to still add the config to the gc config struct. Since there is no repository available there, but I think it still gets us one step closer towards cleaner config.
Quoted 2 lines
Thanks,
Taylor
PATCH v7 8/9 config: make `packed_git_(limit|window_size)` non-global variables
In reply to Karthik NayakThe variables packed_git_window_size and packed_git_limit are global config variables used in the packfile.c file. Since it is only used in this file, let's change it from being a global config variable to a local variable for the subsystem.
With this, we rid packfile.c from all global variable usage and this means we can also remove the USE_THE_REPOSITORY_VARIABLE guard from the file.
--- builtin/fast-import.c | 4 ++-- config.c | 17 ----------------- environment.c | 2 -- packfile.c | 23 +++++++++++++++-------- packfile.h | 2 +- repo-settings.c | 14 ++++++++++++++ repo-settings.h | 5 +++++ 7 files changed, 37 insertions(+), 30 deletions(-)
builtin/fast-import.c +2 -2
@@ -3539,7 +3539,7 @@ static void parse_argv(void)int cmd_fast_import (int argc,const char **argv,const char *prefix, - struct repository *repo UNUSED) +struct repository *repo) {unsigned int i;
@@ -3660,7 +3660,7 @@ int cmd_fast_import(int argc,fprintf (stderr," pools: %10lu KiB\n" , (unsigned long )((tree_entry_allocd + fi_mem_pool.pool_alloc) /1024 ));fprintf (stderr," objects: %10" PRIuMAX" KiB\n" , (alloc_count*sizeof (struct object_entry ))/1024 );fprintf (stderr,"---------------------------------------------------------------------\n" ); -pack_report (); +pack_report (repo);fprintf (stderr,"---------------------------------------------------------------------\n" );fprintf (stderr,"\n" ); }
config.c +0 -17
@@ -1493,28 +1493,11 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.packedgitwindowsize" )) { -int pgsz_x2 =getpagesize () *2 ; - packed_git_window_size =git_config_ulong (var, value, ctx->kvi); - -/* This value must be multiple of (pagesize * 2) */ - packed_git_window_size /= pgsz_x2; -if (packed_git_window_size <1 ) - packed_git_window_size =1 ; - packed_git_window_size *= pgsz_x2; -return 0 ; - } -if (!strcmp (var,"core.bigfilethreshold" )) { big_file_threshold =git_config_ulong (var, value, ctx->kvi);return 0 ; } -if (!strcmp (var,"core.packedgitlimit" )) { - packed_git_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -2
@@ -49,8 +49,6 @@ int fsync_object_files = -1;int use_fsync =-1 ;enum fsync_method fsync_method =FSYNC_METHOD_DEFAULT ;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ; -size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ; -size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
packfile.c +15 -8
@@ -1,4 +1,3 @@ -#define USE_THE_REPOSITORY_VARIABLE #include "git-compat-util.h" #include "environment.h"
@@ -48,15 +47,15 @@ static size_t pack_mapped;#define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; } -void pack_report (void ) +void pack_report (struct repository *repo) {fprintf (stderr,"pack_report: getpagesize() = %10" SZ_FMT "\n" "pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" "pack_report: core.packedGitLimit = %10" SZ_FMT "\n" ,sz_fmt (getpagesize ()), -sz_fmt (packed_git_window_size), -sz_fmt (packed_git_limit)); +sz_fmt (repo->settings.packed_git_window_size), +sz_fmt (repo->settings.packed_git_limit));fprintf (stderr,"pack_report: pack_used_ctr = %10u\n" "pack_report: pack_mmap_calls = %10u\n"
@@ -652,8 +651,15 @@ unsigned char *use_pack(struct packed_git *p,break ; }if (!win) { -size_t window_align = packed_git_window_size /2 ; +size_t window_align;off_t len; +struct repo_settings *settings; + +/* lazy load the settings in case it hasn't been setup */ +prepare_repo_settings (p->repo); + settings = &p->repo->settings; + + window_align = settings->packed_git_window_size /2 ;if (p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name);
@@ -661,11 +667,12 @@ unsigned char *use_pack(struct packed_git *p,CALLOC_ARRAY (win,1 ); win->offset = (offset / window_align) * window_align; len = p->pack_size - win->offset; -if (len > packed_git_window_size) - len = packed_git_window_size; +if (len > settings->packed_git_window_size) + len = settings->packed_git_window_size; win->len = (size_t )len; pack_mapped += win->len; -while (packed_git_limit < pack_mapped + +while (settings->packed_git_limit < pack_mapped &&unuse_one_window (p)) ;/* nothing */ win->base = xmmap_gently(NULL, win->len,
packfile.h +1 -1
@@ -89,7 +89,7 @@ unsigned long repo_approximate_object_count(struct repository *r);struct packed_git *find_oid_pack (const struct object_id *oid,struct packed_git *packs); -void pack_report (void ); +void pack_report (struct repository *repo);/* * mmap the index file for the specified packfile (if it is not
repo-settings.c +14 -0
@@ -26,6 +26,7 @@ void prepare_repo_settings(struct repository *r)const char *strval;int manyfiles;int read_changed_paths; +unsigned long longval;if (!r->gitdir)BUG ("Cannot add settings for uninitialized repository" );
@@ -123,6 +124,19 @@ void prepare_repo_settings(struct repository *r)* removed. */ r->settings.command_requires_full_index =1 ; + +if (!repo_config_get_ulong (r,"core.packedgitwindowsize" , &longval)) { +int pgsz_x2 =getpagesize () *2 ; + +/* This value must be multiple of (pagesize * 2) */ + longval /= pgsz_x2; +if (longval <1 ) + longval =1 ; + r->settings.packed_git_window_size = longval * pgsz_x2; + } + +if (!repo_config_get_ulong (r,"core.packedgitlimit" , &longval)) + r->settings.packed_git_limit = longval; }enum log_refs_config repo_settings_get_log_all_ref_updates (struct repository *repo)
repo-settings.h +5 -0
@@ -57,12 +57,17 @@ struct repo_settings {int core_multi_pack_index;int warn_ambiguous_refs;/* lazily loaded via accessor */ + +size_t packed_git_window_size; +size_t packed_git_limit; };#define REPO_SETTINGS_INIT { \ .index_version = -1, \ .core_untracked_cache = UNTRACKED_CACHE_KEEP, \ .fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \ .warn_ambiguous_refs = -1, \ + .packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE, \ + .packed_git_limit = DEFAULT_PACKED_GIT_LIMIT, \ }void prepare_repo_settings (struct repository *r);
Signature · 1 line
2.47.0
PATCH v7 9/9 midx: add repository to `multi_pack_index` struct
In reply to Karthik NayakThe multi_pack_index struct represents the MIDX for a repository. Here, we add a pointer to the repository in this struct, allowing direct use of the repository variable without relying on the global the_repository struct.
With this addition, we can determine the repository associated with a bitmap_index struct. A bitmap_index points to either a packed_git or a multi_pack_index, both of which have direct repository references. To support this, we introduce a static helper function, bitmap_repo, in pack-bitmap.c, which retrieves a repository given a bitmap_index.
With this, we clear up all usages of the_repository within pack-bitmap.c and also remove the USE_THE_REPOSITORY_VARIABLE definition. Bringing us another step closer to remove all global variable usage.
Although this change also opens up the potential to clean up midx.c, doing so would require additional refactoring to pass the repository struct to functions where the MIDX struct is created: a task better suited for future patches.
--- midx.c | 1 + midx.h | 3 ++ pack-bitmap.c | 90 +++++++++++++++++++++++++++++++-------------------- 3 files changed, 59 insertions(+), 35 deletions(-)
midx.c +1 -0
@@ -131,6 +131,7 @@ static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir m->data = midx_map; m->data_len = midx_size; m->local = local; + m->repo = the_repository; m->signature =get_be32 (m->data);if (m->signature !=MIDX_SIGNATURE )
midx.h +3 -0
@@ -71,6 +71,9 @@ struct multi_pack_index {const char **pack_names;struct packed_git **packs; + +struct repository *repo; +char object_dir[FLEX_ARRAY ]; };
pack-bitmap.c +55 -35
251 lines not on this page · raw message
Signature · 1 line
2.47.0
Quoted 4 lines · Karthik Nayak
Changes in v7:
- Cleanup stale commit message.
- Add missing space inifstatement.
- Fix typo s/incase/in case/.
Thanks, I think this addresses all of the comments I had on previous versions.
-Peff
Quoted 10 lines · Jeff King
Quoted 4 lines · Karthik Nayak
Changes in v7:
- Cleanup stale commit message.
- Add missing space inifstatement.
- Fix typo s/incase/in case/.Thanks, I think this addresses all of the comments I had on previous versions.
Thanks for your review!
- Karthik
Quoted 10 lines · Karthik Nayak
Karthik Nayak (9):
packfile: add repository to structpacked_git
packfile: userepositoryfrompacked_gitdirectly
packfile: passrepositoryto static function in the file
packfile: pass down repository toodb_pack_name
packfile: pass down repository tohas_object[_kept]_pack
packfile: pass down repository tofor_each_packed_object
config: makedelta_base_cache_limita non-global variable
config: makepacked_git_(limit|window_size)non-global variables
midx: add repository tomulti_pack_indexstruct
I reviewed this round, and think that it is looking very close. There are a couple of typofixes that I and others have noticed, which are minor (but I think in aggregate should merit a reroll).
I did have a concern about the conversion of delta_base_cache_limit to be a non-global variable, since I think we're determining that value from within unpack_entry() in a more expensive manner than is possible.
So I think that merits some investigation, and will likely result in some changes that we should consider before merging.
Karthik: if you do end up rerolling this, please feel free to include the patch I sent in [1] on top, which should make the maintainer's life a bit easier than adding another topic dependent upon this one ;-).
Thanks,
Taylor
[1]: https://lore.kernel.org/git/884ca9770d1fb1e84962b1f700b1ce4adce6321c.1732142889.git.me@ttaylorr.com/
Quoted 12 lines · Taylor Blau
Quoted 10 lines · Karthik Nayak
Karthik Nayak (9):
packfile: add repository to structpacked_git
packfile: userepositoryfrompacked_gitdirectly
packfile: passrepositoryto static function in the file
packfile: pass down repository toodb_pack_name
packfile: pass down repository tohas_object[_kept]_pack
packfile: pass down repository tofor_each_packed_object
config: makedelta_base_cache_limita non-global variable
config: makepacked_git_(limit|window_size)non-global variables
midx: add repository tomulti_pack_indexstruct
Quoted 4 lines
I reviewed this round, and think that it is looking very close. There are a couple of typofixes that I and others have noticed, which are minor (but I think in aggregate should merit a reroll).
Thanks for the review, indeed, I will be re-rolling with the fixes discussed.
Quoted 7 lines
I did have a concern about the conversion of
delta_base_cache_limitto be a non-global variable, since I think we're determining that value from withinunpack_entry()in a more expensive manner than is possible.So I think that merits some investigation, and will likely result in some changes that we should consider before merging.
Yup, I'll move it to the repository settings struct and this should help alleviate the perf issue.
Quoted 4 lines
Karthik: if you do end up rerolling this, please feel free to include the patch I sent in [1] on top, which should make the maintainer's life a bit easier than adding another topic dependent upon this one ;-).
That makes sense, I'll add it in.
Quoted 4 lines
Thanks,
Taylor[1]: https://lore.kernel.org/git/884ca9770d1fb1e84962b1f700b1ce4adce6321c.1732142889.git.me@ttaylorr.com/
v8 cover packfile: avoid using the 'the_repository' global variable
In reply to Karthik NayakThe packfile.c file uses the global variable 'the_repository' extensively throughout the code. Let's remove all usecases of this, by modifying the required functions to accept a 'struct repository' instead. This is to clean up usage of global state.
The first 3 patches are mostly internal to packfile.c, we add the repository field to the packed_git struct and this is used to clear up some useages of the global variables.
The next 3 patches are more disruptive, they modify the function definition of odb_pack_name, has_object[_kept]_pack and for_each_packed_object to receive a repository, helping remove other usages of 'the_repository' variable.
Finally, the next two patches deal with global config values. These values are localized. The last patch is removal of an unecessary call to prepare_packed_git(). For v5 onwards, I've rebased the series off the master: 8f8d6eee53 (The seventh batch, 2024-11-01), as a dependency for this series 'jk/dumb-http-finalize' was merged to master. I've found no conflicts while merging with seen & next. But since this series does touch multiple files, there could be future conflicts.
Changes in v8:
- Fix typos in comments
- For packfile.c use delta_base_cache_limit from the repository settings, this avoids loading the config in hot paths.
- Rename longval to ulongval to better signify the type.
Changes in v7:
- Cleanup stale commit message.
- Add missing space in if statement.
- Fix typo s/incase/in case/.
Changes in v6:
- Lazy load repository settings in packfile.c. This ensures that the settings are available for sure and we do not rely on callees setting it.
- Use size_t for delta_base_cache_limit.
Changes in v5:
- Move packed_git* settings to repo_settings to ensure we don't keep reparsing the settings in use_pack.
Changes in v4:
- Renamed the repository field within packed_git and multi_pack_index from r to repo, while keeping function parameters to be r.
- Fixed bad braces.
Changes in v3:
- Improved commit messages. In the first commit to talk about how packed_git struct could also be part of the alternates of a repository. In the 7th commit to talk about the motive behind removing the global variable.
- Changed 'packed_git->repo' to 'packed_git->r' to keep it consistent with the rest of the code base.
- Replaced 'the_repository' with locally available access to the repository struct in multiple regions.
- Removed unecessary inclusion of the 'repository.h' header file by forward declaring the 'repository' struct.
- Replace memcpy with hashcpy.
- Change the logic in the 7th patch to use if else statements.
- Added an extra commit to cleanup pack-bitmap.c.
Karthik Nayak (9):
packfile: add repository to struct packed_git
packfile: use repository from packed_git directly
packfile: pass repository to static function in the file
packfile: pass down repository to odb_pack_name
packfile: pass down repository to has_object[_kept]_pack
packfile: pass down repository to for_each_packed_object
config: make delta_base_cache_limit a non-global variable
config: make packed_git_(limit|window_size) non-global variables
midx: add repository to multi_pack_index struct
Taylor Blau (1):
packfile.c: remove unnecessary prepare_packed_git() call
builtin/cat-file.c | 7 +- builtin/count-objects.c | 2 +- builtin/fast-import.c | 15 ++-- builtin/fsck.c | 20 +++--- builtin/gc.c | 8 ++- builtin/index-pack.c | 20 ++++-- builtin/pack-objects.c | 11 +-- builtin/pack-redundant.c | 2 +- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 4 +- config.c | 22 ------ connected.c | 3 +- diff.c | 3 +- environment.c | 3 - environment.h | 1 - fsck.c | 2 +- http.c | 4 +- list-objects.c | 7 +- midx-write.c | 2 +- midx.c | 3 +- midx.h | 3 + object-store-ll.h | 9 ++- pack-bitmap.c | 90 ++++++++++++++---------- pack-objects.h | 3 +- pack-write.c | 1 + pack.h | 2 + packfile.c | 144 ++++++++++++++++++++++----------------- packfile.h | 18 +++-- promisor-remote.c | 2 +- prune-packed.c | 2 +- reachable.c | 4 +- repo-settings.c | 18 +++++ repo-settings.h | 7 ++ revision.c | 13 ++-- tag.c | 2 +- 36 files changed, 271 insertions(+), 190 deletions(-)
Range-diff against v7:
1: 6c00e25c86 ! 1: d1fdd6996a packfile: add repository to struct packed_git
@@ object-store-ll.h: struct packed_git {
const uint32_t *mtimes_map;
size_t mtimes_size;
+
-+ /* repo dentoes the repository this packed file belongs to */
++ /* repo denotes the repository this packfile belongs to */
+ struct repository *repo;
+
/* something like ".git/objects/pack/xxxxx.pack" */
2: 70fc8a79af = 2: 65c09858ce packfile: use `repository` from `packed_git` directly
3: 167a1f3a11 = 3: 80632934d1 packfile: pass `repository` to static function in the file
4: b7cfe78217 = 4: 67d71eab83 packfile: pass down repository to `odb_pack_name`
5: 5566f5554c = 5: ee210fa153 packfile: pass down repository to `has_object[_kept]_pack`
6: 1b26e45a9b = 6: 8db7094f4e packfile: pass down repository to `for_each_packed_object`
7: 1bdc34f4d8 ! 7: d1b6e8801b config: make `delta_base_cache_limit` a non-global variable@@ Commit message
The delta_base_cache_limit variable is a global config variable used by multiple subsystems. Let's make this non-global, by adding this
- variable to the stack of each of the subsystems where it is used.
+ variable independently to the subsystems where it is used.
- In gc.c we add it to the gc_config struct and also the constructor
- function. In index-pack.c we add it to the pack_idx_option struct
- and its constructor. Finally, in packfile.c we dynamically retrieve
- this value from the repository config, since the value is only used once
- in the entire subsystem.
+ First, add the setting to the repo_settings struct, this provides
+ access to the config in places where the repository is available. Use
+ this in packfile.c.
+
+ In index-pack.c we add it to the pack_idx_option struct and its
+ constructor. While the repository struct is available here, it may not
+ be set because git index-pack can be used without a repository.
+
+ In gc.c add it to the gc_config struct and also the constructor
+ function. The gc functions currently do not have direct access to a
+ repository struct.
These changes are made to remove the usage of delta_base_cache_limit as a global variable in packfile.c. This brings us one step closer to
@@ pack.h: struct pack_idx_option {
*/
int anomaly_alloc, anomaly_nr;
uint32_t *anomaly;
-+ unsigned long delta_base_cache_limit;
++
++ size_t delta_base_cache_limit;
};
void reset_pack_idx_option(struct pack_idx_option *);
## packfile.c ##
-@@
- #include "commit-graph.h"
- #include "pack-revindex.h"
- #include "promisor-remote.h"
-+#include "config.h"
-+#include "pack-objects.h"
-
- char *odb_pack_name(struct repository *r, struct strbuf *buf,
- const unsigned char *hash, const char *ext)
@@ packfile.c: void clear_delta_base_cache(void)
}
@@ packfile.c: void clear_delta_base_cache(void)
struct delta_base_cache_entry *ent;
struct list_head *lru, *tmp;
@@ packfile.c: void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,
- struct unpack_entry_stack_ent *delta_stack = small_delta_stack;
int delta_stack_nr = 0, delta_stack_alloc = UNPACK_ENTRY_STACK_PREALLOC;
int base_from_cache = 0;
-+ unsigned long delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT;
-+
-+ repo_config_get_ulong(r, "core.deltabasecachelimit", &delta_base_cache_limit);
++ prepare_repo_settings(p->repo);
++
write_pack_access_log(p, obj_offset);
+ /* PHASE 1: drill down to the innermost base object */
@@ packfile.c: void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,
* before we are done using it.
*/
if (!external_base)
- add_delta_base_cache(p, base_obj_offset, base, base_size, type);
-+ add_delta_base_cache(p, base_obj_offset, base,
-+ base_size, delta_base_cache_limit,
++ add_delta_base_cache(p, base_obj_offset, base, base_size,
++ p->repo->settings.delta_base_cache_limit,
+ type);
free(delta_data);
free(external_base);
+
+ ## repo-settings.c ##
+@@
+ #include "repo-settings.h"
+ #include "repository.h"
+ #include "midx.h"
++#include "pack-objects.h"
+
+ static void repo_cfg_bool(struct repository *r, const char *key, int *dest,
+ int def)
+@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
+ const char *strval;
+ int manyfiles;
+ int read_changed_paths;
++ unsigned long ulongval;
+
+ if (!r->gitdir)
+ BUG("Cannot add settings for uninitialized repository");
+@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
+ * removed.
+ */
+ r->settings.command_requires_full_index = 1;
++
++ if (!repo_config_get_ulong(r, "core.deltabasecachelimit", &ulongval))
++ r->settings.delta_base_cache_limit = ulongval;
+ }
+
+ enum log_refs_config repo_settings_get_log_all_ref_updates(struct repository *repo) +
+ ## repo-settings.h ##
+@@ repo-settings.h: struct repo_settings {
+
+ int core_multi_pack_index;
+ int warn_ambiguous_refs; /* lazily loaded via accessor */
++
++ size_t delta_base_cache_limit;
+ };
+ #define REPO_SETTINGS_INIT { \
+ .index_version = -1, \
+ .core_untracked_cache = UNTRACKED_CACHE_KEEP, \
+ .fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \
+ .warn_ambiguous_refs = -1, \
++ .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \
+ }
+
+ void prepare_repo_settings(struct repository *r);
8: 7b6baa89ac ! 8: 30a52f192f config: make packed_git_(limit|window_size) non-global variables
@@ packfile.h: unsigned long repo_approximate_object_count(struct repository *r);
## repo-settings.c ##
@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
- const char *strval;
- int manyfiles;
- int read_changed_paths;
-+ unsigned long longval;
- if (!r->gitdir)
- BUG("Cannot add settings for uninitialized repository");
-@@ repo-settings.c: void prepare_repo_settings(struct repository *r)
- * removed.
- */
- r->settings.command_requires_full_index = 1;
+ if (!repo_config_get_ulong(r, "core.deltabasecachelimit", &ulongval))
+ r->settings.delta_base_cache_limit = ulongval;
+
-+ if (!repo_config_get_ulong(r, "core.packedgitwindowsize", &longval)) {
++ if (!repo_config_get_ulong(r, "core.packedgitwindowsize", &ulongval)) {
+ int pgsz_x2 = getpagesize() * 2;
+
+ /* This value must be multiple of (pagesize * 2) */
-+ longval /= pgsz_x2;
-+ if (longval < 1)
-+ longval = 1;
-+ r->settings.packed_git_window_size = longval * pgsz_x2;
++ ulongval /= pgsz_x2;
++ if (ulongval < 1)
++ ulongval = 1;
++ r->settings.packed_git_window_size = ulongval * pgsz_x2;
+ }
+
-+ if (!repo_config_get_ulong(r, "core.packedgitlimit", &longval))
-+ r->settings.packed_git_limit = longval;
++ if (!repo_config_get_ulong(r, "core.packedgitlimit", &ulongval))
++ r->settings.packed_git_limit = ulongval;
}
enum log_refs_config repo_settings_get_log_all_ref_updates(struct repository *repo)
## repo-settings.h ##
@@ repo-settings.h: struct repo_settings {
-
- int core_multi_pack_index;
int warn_ambiguous_refs; /* lazily loaded via accessor */
-+
+
+ size_t delta_base_cache_limit;
+ size_t packed_git_window_size;
+ size_t packed_git_limit;
};
#define REPO_SETTINGS_INIT { \
.index_version = -1, \
- .core_untracked_cache = UNTRACKED_CACHE_KEEP, \
+@@ repo-settings.h: struct repo_settings {
.fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \
.warn_ambiguous_refs = -1, \
+ .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \
+ .packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE, \
+ .packed_git_limit = DEFAULT_PACKED_GIT_LIMIT, \
}
9: a3667d87ec = 9: 2fe5d2506f midx: add repository to `multi_pack_index` struct
-: ---------- > 10: 05989c2e27 packfile.c: remove unnecessary prepare_packed_git() callSignature · 1 line
2.47.0
PATCH v8 1/10 packfile: add repository to struct `packed_git`
The struct packed_git holds information regarding a packed object file. Let's add the repository variable to this object, to represent the repository that this packfile belongs to. This helps remove dependency on the global the_repository object in packfile.c by simply using repository information now readily available in the struct.
We do need to consider that a pack file could be part of the alternates of a repository, but considering that we only have one repository struct and also that we currently anyways use 'the_repository'. We should be OK with this change.
We also modify alloc_packed_git to ensure that the repository is added to newly created packed_git structs. This requires modifying the function and all its callee to pass the repository object down the levels.
--- builtin/fast-import.c | 3 ++- builtin/index-pack.c | 6 ++++-- commit-graph.c | 2 +- connected.c | 3 ++- http.c | 2 +- midx-write.c | 2 +- midx.c | 2 +- object-store-ll.h | 5 +++++ packfile.c | 15 +++++++++------ packfile.h | 6 ++++-- 10 files changed, 30 insertions(+), 16 deletions(-)
builtin/fast-import.c +2 -1
@@ -765,6 +765,7 @@ static void start_packfile(void) p->pack_fd = pack_fd; p->do_not_close =1 ; + p->repo = the_repository; pack_file =hashfd (pack_fd, p->pack_name); pack_data = p;
@@ -888,7 +889,7 @@ static void end_packfile(void) idx_name =keep_pack (create_index ());/* Register the packfile with core git's machinery. */ - new_p =add_packed_git (idx_name,strlen (idx_name),1 ); + new_p =add_packed_git (pack_data->repo, idx_name,strlen (idx_name),1 );if (!new_p)die ("core git rejected index %s" , idx_name); all_packs[pack_id] = new_p;
builtin/index-pack.c +4 -2
@@ -1552,7 +1552,8 @@ static void final(const char *final_pack_name, const char *curr_pack_name,if (do_fsck_object) {struct packed_git *p; - p =add_packed_git (final_index_name,strlen (final_index_name),0 ); + p =add_packed_git (the_repository, final_index_name, +strlen (final_index_name),0 );if (p)install_packed_git (the_repository, p); }
@@ -1650,7 +1651,8 @@ static void read_v2_anomalous_offsets(struct packed_git *p,static void read_idx_option (struct pack_idx_option *opts,const char *pack_name) { -struct packed_git *p =add_packed_git (pack_name,strlen (pack_name),1 ); +struct packed_git *p =add_packed_git (the_repository, pack_name, +strlen (pack_name),1 );if (!p)die (_ ("Cannot open existing pack file '%s'" ), pack_name);
commit-graph.c +1 -1
@@ -1914,7 +1914,7 @@ static int fill_oids_from_packs(struct write_commit_graph_context *ctx,struct packed_git *p;strbuf_setlen (&packname, dirlen);strbuf_addstr (&packname, pack_indexes->items[i].string); - p =add_packed_git (packname.buf, packname.len,1 ); + p =add_packed_git (ctx->r, packname.buf, packname.len,1 );if (!p) { ret =error (_ ("error adding pack %s" ), packname.buf); goto cleanup;
connected.c +2 -1
@@ -54,7 +54,8 @@ int check_connected(oid_iterate_fn fn, void *cb_data,strbuf_add (&idx_file, transport->pack_lockfiles.items[0 ].string, base_len);strbuf_addstr (&idx_file,".idx" ); - new_pack =add_packed_git (idx_file.buf, idx_file.len,1 ); + new_pack =add_packed_git (the_repository, idx_file.buf, + idx_file.len,1 );strbuf_release (&idx_file); }
http.c +1 -1
@@ -2439,7 +2439,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,if (!tmp_idx)return -1 ; - new_pack =parse_pack_index (sha1, tmp_idx); + new_pack =parse_pack_index (the_repository, sha1, tmp_idx);if (!new_pack) {unlink (tmp_idx);free (tmp_idx);
midx-write.c +1 -1
@@ -154,7 +154,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,return ;ALLOC_GROW (ctx->info, ctx->nr +1 , ctx->alloc); - p =add_packed_git (full_path, full_path_len,0 ); + p =add_packed_git (the_repository, full_path, full_path_len,0 );if (!p) {warning (_ ("failed to add packfile '%s'" ), full_path);
midx.c +1 -1
@@ -464,7 +464,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,strhash (key.buf), key.buf, struct packed_git, packmap_ent);if (!p) { - p =add_packed_git (pack_name.buf, pack_name.len, m->local); + p =add_packed_git (r, pack_name.buf, pack_name.len, m->local);if (p) {install_packed_git (r, p);list_add_tail (&p->mru, &r->objects->packed_git_mru);
object-store-ll.h +5 -0
@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;struct object_directory {struct object_directory *next;
@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo denotes the repository this packfile belongs to */ +struct repository *repo; +/* something like ".git/objects/pack/xxxxx.pack" */ char pack_name[FLEX_ARRAY ];/* more */ };
packfile.c +9 -6
@@ -217,11 +217,12 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)return ntohl (level1_ofs[value]); } -static struct packed_git *alloc_packed_git (int extra) +static struct packed_git *alloc_packed_git (struct repository *r,int extra) {struct packed_git *p =xmalloc (st_add (sizeof (*p), extra));memset (p,0 ,sizeof (*p)); p->pack_fd =-1 ; + p->repo = r;return p; }
@@ -233,11 +234,12 @@ static char *pack_path_from_idx(const char *idx_path)return xstrfmt ("%.*s.pack" , (int )len, idx_path); } -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path) {char *path =pack_path_from_idx (idx_path);size_t alloc =st_add (strlen (path),1 ); -struct packed_git *p =alloc_packed_git (alloc); +struct packed_git *p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path);
@@ -703,7 +705,8 @@ void unuse_pack(struct pack_window **w_cursor) } } -struct packed_git *add_packed_git (const char *path,size_t path_len,int local) +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local) {struct stat st;size_t alloc;
@@ -721,7 +724,7 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)* the use xsnprintf double-checks that) */ alloc =st_add3 (path_len,strlen (".promisor" ),1 ); - p =alloc_packed_git (alloc); + p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, path_len);xsnprintf (p->pack_name + path_len, alloc - path_len,".keep" );
@@ -877,7 +880,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,/* Don't reopen a pack we already have. */ if (!hashmap_get (&data->r->objects->pack_map, &hent, pack_name)) { - p =add_packed_git (full_name, full_name_len, data->local); + p =add_packed_git (data->r, full_name, full_name_len, data->local);if (p)install_packed_git (data->r, p); }
packfile.h +4 -2
@@ -46,7 +46,8 @@ const char *pack_basename(struct packed_git *p);* and does not add the resulting packed_git struct to the internal list of * packs. You probably want add_packed_git() instead. */ -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path);typedef void each_file_in_pack_dir_fn (const char *full_path,size_t full_path_len,const char *file_name,void *data);
@@ -113,7 +114,8 @@ void close_pack(struct packed_git *);void close_object_store (struct raw_object_store *o);void unuse_pack (struct pack_window **);void clear_delta_base_cache (void ); -struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local);/* * Unlink the .pack and associated extension files.
Signature · 1 line
2.47.0
PATCH v8 2/10 packfile: use `repository` from `packed_git` directly
In reply to Karthik NayakIn the previous commit, we introduced the repository structure inside packed_git. This provides an alternative route instead of using the global the_repository variable. Let's modify packfile.c now to use this field wherever possible instead of relying on the global state. There are still a few instances of the_repository usage in the file, where there is no struct packed_git locally available, which will be fixed in the following commits.
--- packfile.c | 50 +++++++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 23 deletions(-)
packfile.c +27 -23
@@ -79,7 +79,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)size_t idx_size;int fd =git_open (path), ret;struct stat st; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (fd <0 )return -1 ;
@@ -243,7 +243,7 @@ struct packed_git *parse_pack_index(struct repository *r, unsigned char *sha1,memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path); -hashcpy (p->hash, sha1, the_repository->hash_algo); +hashcpy (p->hash, sha1, p->repo->hash_algo);if (check_packed_git_idx (idx_path, p)) {free (p);return NULL ;
@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = the_repository->objects->packed_git; p; p = p->next) +for (p = current->repo->objects->packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -540,7 +540,7 @@ static int open_packed_git_1(struct packed_git *p)unsigned char hash[GIT_MAX_RAWSZ ];unsigned char *idx_hash;ssize_t read_result; -const unsigned hashsz = the_hash_algo->rawsz; +const unsigned hashsz = p->repo->hash_algo->rawsz;if (open_pack_index (p))return error ("packfile %s index unavailable" , p->pack_name);
@@ -597,7 +597,7 @@ static int open_packed_git_1(struct packed_git *p)if (read_result != hashsz)return error ("packfile %s signature is unavailable" , p->pack_name); idx_hash = ((unsigned char *)p->index_data) + p->index_size - hashsz *2 ; -if (!hasheq (hash, idx_hash, the_repository->hash_algo)) +if (!hasheq (hash, idx_hash, p->repo->hash_algo))return error ("packfile %s does not match index" , p->pack_name);return 0 ; }
@@ -637,7 +637,7 @@ unsigned char *use_pack(struct packed_git *p,*/ if (!p->pack_size && p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name); -if (offset > (p->pack_size - the_hash_algo->rawsz)) +if (offset > (p->pack_size - p->repo->hash_algo->rawsz))die ("offset beyond end of packfile (truncated pack?)" );if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" ));
@@ -711,6 +711,7 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,struct stat st;size_t alloc;struct packed_git *p; +struct object_id oid;/* * Make sure a corresponding .pack file exists and that
@@ -751,9 +752,13 @@ struct packed_git *add_packed_git(struct repository *r, const char *path, p->pack_size = st.st_size; p->pack_local = local; p->mtime = st.st_mtime; -if (path_len < the_hash_algo->hexsz || -get_hash_hex (path + path_len - the_hash_algo->hexsz, p->hash)) -hashclr (p->hash, the_repository->hash_algo); +if (path_len < r->hash_algo->hexsz || +get_oid_hex_algop (path + path_len - r->hash_algo->hexsz, &oid, + r->hash_algo)) +hashclr (p->hash, r->hash_algo); +else +hashcpy (p->hash, oid.hash, r->hash_algo); +return p; }
@@ -1243,9 +1248,9 @@ off_t get_delta_base(struct packed_git *p, }else if (type ==OBJ_REF_DELTA ) {/* The base entry _must_ be in the same pack */ struct object_id oid; -oidread (&oid, base_info, the_repository->hash_algo); +oidread (&oid, base_info, p->repo->hash_algo); base_offset =find_pack_entry_one (&oid, p); - *curpos += the_hash_algo->rawsz; + *curpos += p->repo->hash_algo->rawsz; }else die ("I am totally screwed" );return base_offset;
@@ -1266,7 +1271,7 @@ static int get_delta_base_oid(struct packed_git *p, {if (type ==OBJ_REF_DELTA ) {unsigned char *base =use_pack (p, w_curs, curpos,NULL ); -oidread (oid, base, the_repository->hash_algo); +oidread (oid, base, p->repo->hash_algo);return 0 ; }else if (type ==OBJ_OFS_DELTA ) {uint32_t base_pos;
@@ -1608,7 +1613,7 @@ int packed_object_info(struct repository *r, struct packed_git *p, goto out; } }else -oidclr (oi->delta_base_oid, the_repository->hash_algo); +oidclr (oi->delta_base_oid, p->repo->hash_algo); } oi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :
@@ -1897,7 +1902,7 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32 {const unsigned char *index_fanout = p->index_data;const unsigned char *index_lookup; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;int index_lookup_width;if (!index_fanout)
@@ -1922,7 +1927,7 @@ int nth_packed_object_id(struct object_id *oid, uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (!index) {if (open_pack_index (p))return -1 ;
@@ -1933,11 +1938,10 @@ int nth_packed_object_id(struct object_id *oid, index +=4 *256 ;if (p->index_version ==1 ) {oidread (oid, index +st_add (st_mult (hashsz +4 , n),4 ), - the_repository->hash_algo); + p->repo->hash_algo); }else { index +=8 ; -oidread (oid, index +st_mult (hashsz, n), - the_repository->hash_algo); +oidread (oid, index +st_mult (hashsz, n), p->repo->hash_algo); }return 0 ; }
@@ -1959,7 +1963,7 @@ void check_pack_index_ptr(const struct packed_git *p, const void *vptr)off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz; index +=4 *256 ;if (p->index_version ==1 ) {return ntohl (*((uint32_t *)(index +st_mult (hashsz +4 , n))));
@@ -2159,7 +2163,7 @@ int for_each_object_in_pack(struct packed_git *p,int r =0 ;if (flags &FOR_EACH_OBJECT_PACK_ORDER ) { -if (load_pack_revindex (the_repository, p)) +if (load_pack_revindex (p->repo, p))return -1 ; }
@@ -2227,7 +2231,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, }static int add_promisor_object (const struct object_id *oid, - struct packed_git *pack UNUSED, +struct packed_git *pack, uint32_t pos UNUSED,void *set_) {
@@ -2235,12 +2239,12 @@ static int add_promisor_object(const struct object_id *oid,struct object *obj;int we_parsed_object; - obj =lookup_object (the_repository, oid); + obj =lookup_object (pack->repo, oid);if (obj && obj->parsed) { we_parsed_object =0 ; }else { we_parsed_object =1 ; - obj =parse_object (the_repository, oid); + obj =parse_object (pack->repo, oid); }if (!obj)
Signature · 1 line
2.47.0
PATCH v8 3/10 packfile: pass `repository` to static function in the file
In reply to Karthik NayakSome of the static functions in the packfile.c access global variables, which can simply be avoiding by passing the repository struct down to them. Let's do that.
--- packfile.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
packfile.c +8 -7
@@ -460,13 +460,13 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc *accept_windows_inuse = has_windows_inuse; } -static int close_one_pack (void ) +static int close_one_pack (struct repository *r) {struct packed_git *p, *lru_p =NULL ;struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = the_repository->objects->packed_git; p; p = p->next) { +for (p = r->objects->packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -555,7 +555,7 @@ static int open_packed_git_1(struct packed_git *p) pack_max_fds =1 ; } -while (pack_max_fds <= pack_open_fds &&close_one_pack ()) +while (pack_max_fds <= pack_open_fds &&close_one_pack (p->repo)) ;/* nothing */ p->pack_fd =git_open (p->pack_name);
@@ -610,7 +610,8 @@ static int open_packed_git(struct packed_git *p)return -1 ; } -static int in_window (struct pack_window *win,off_t offset) +static int in_window (struct repository *r,struct pack_window *win, +off_t offset) {/* We must promise at least one full hash after the * offset is available from this window, otherwise the offset
@@ -620,7 +621,7 @@ static int in_window(struct pack_window *win, off_t offset)*/ off_t win_off = win->offset;return win_off <= offset - && (offset + the_hash_algo->rawsz) <= (win_off + win->len); + && (offset + r->hash_algo->rawsz) <= (win_off + win->len); } unsigned char *use_pack(struct packed_git *p,
@@ -642,11 +643,11 @@ unsigned char *use_pack(struct packed_git *p,if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" )); -if (!win || !in_window (win, offset)) { +if (!win || !in_window (p->repo, win, offset)) {if (win) win->inuse_cnt--;for (win = p->windows; win; win = win->next) { -if (in_window (win, offset)) +if (in_window (p->repo, win, offset))break ; }if (!win) {
Signature · 1 line
2.47.0
PATCH v8 4/10 packfile: pass down repository to `odb_pack_name`
In reply to Karthik NayakThe function odb_pack_name currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers.
--- builtin/fast-import.c | 8 ++++---- builtin/index-pack.c | 4 ++-- builtin/pack-redundant.c | 2 +- http.c | 2 +- packfile.c | 9 ++++----- packfile.h | 3 ++- 6 files changed, 14 insertions(+), 14 deletions(-)
builtin/fast-import.c +4 -4
@@ -806,7 +806,7 @@ static char *keep_pack(const char *curr_index_name)struct strbuf name =STRBUF_INIT ;int keep_fd; -odb_pack_name (&name, pack_data->hash,"keep" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"keep" ); keep_fd =odb_pack_keep (name.buf);if (keep_fd <0 )die_errno ("cannot create keep file" );
@@ -814,11 +814,11 @@ static char *keep_pack(const char *curr_index_name)if (close (keep_fd))die_errno ("failed to write keep file" ); -odb_pack_name (&name, pack_data->hash,"pack" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"pack" );if (finalize_object_file (pack_data->pack_name, name.buf))die ("cannot store pack file" ); -odb_pack_name (&name, pack_data->hash,"idx" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"idx" );if (finalize_object_file (curr_index_name, name.buf))die ("cannot store index file" );free ((void *)curr_index_name);
@@ -832,7 +832,7 @@ static void unkeep_all_packs(void)for (k =0 ; k < pack_id; k++) {struct packed_git *p = all_packs[k]; -odb_pack_name (&name, p->hash,"keep" ); +odb_pack_name (p->repo, &name, p->hash,"keep" );unlink_or_warn (name.buf); }strbuf_release (&name);
builtin/index-pack.c +2 -2
@@ -1479,7 +1479,7 @@ static void write_special_file(const char *suffix, const char *msg,if (pack_name) filename =derive_filename (pack_name,"pack" , suffix, &name_buf);else - filename =odb_pack_name (&name_buf, hash, suffix); + filename =odb_pack_name (the_repository, &name_buf, hash, suffix); fd =odb_pack_keep (filename);if (fd <0 ) {
@@ -1507,7 +1507,7 @@ static void rename_tmp_packfile(const char **final_name, {if (!*final_name ||strcmp (*final_name, curr_name)) {if (!*final_name) - *final_name =odb_pack_name (name, hash, ext); + *final_name =odb_pack_name (the_repository, name, hash, ext);if (finalize_object_file (curr_name, *final_name))die (_ ("unable to rename temporary '*.%s' file to '%s'" ), ext, *final_name);
builtin/pack-redundant.c +1 -1
@@ -690,7 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix UNUSED, s pl = red =pack_list_difference (local_packs, min);while (pl) {printf ("%s\n%s\n" , -odb_pack_name (&idx_name, pl->pack->hash,"idx" ), +odb_pack_name (pl->pack->repo, &idx_name, pl->pack->hash,"idx" ), pl->pack->pack_name); pl = pl->next; }
http.c +1 -1
@@ -2581,7 +2581,7 @@ struct http_pack_request *new_direct_http_pack_request( preq->url = url; -odb_pack_name (&preq->tmpfile, packed_git_hash,"pack" ); +odb_pack_name (the_repository, &preq->tmpfile, packed_git_hash,"pack" );strbuf_addstr (&preq->tmpfile,".temp" ); preq->packfile =fopen (preq->tmpfile.buf,"a" );if (!preq->packfile) {
packfile.c +4 -5
@@ -25,13 +25,12 @@#include "pack-revindex.h" #include "promisor-remote.h" -char *odb_pack_name (struct strbuf *buf, -const unsigned char *hash, -const char *ext) +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext) {strbuf_reset (buf); -strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (the_repository), -hash_to_hex (hash), ext); +strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (r), +hash_to_hex_algop (hash, r->hash_algo), ext);return buf->buf; }
packfile.h +2 -1
@@ -29,7 +29,8 @@ struct pack_entry {* * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" */ -char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext);/* * Return the basename of the packfile, omitting any containing directory
Signature · 1 line
2.47.0
PATCH v8 5/10 packfile: pass down repository to `has_object[_kept]_pack`
In reply to Karthik NayakThe functions has_object[_kept]_pack currently rely on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from these functions and any related ones.
--- builtin/count-objects.c | 2 +- builtin/fsck.c | 2 +- builtin/pack-objects.c | 4 ++-- diff.c | 3 ++- list-objects.c | 3 ++- pack-bitmap.c | 2 +- packfile.c | 9 +++++---- packfile.h | 5 +++-- prune-packed.c | 2 +- reachable.c | 2 +- revision.c | 4 ++-- 11 files changed, 21 insertions(+), 17 deletions(-)
builtin/count-objects.c +1 -1
@@ -67,7 +67,7 @@ static int count_loose(const struct object_id *oid, const char *path,else { loose_size +=on_disk_bytes (st); loose++; -if (verbose &&has_object_pack (oid)) +if (verbose &&has_object_pack (the_repository, oid)) packed_loose++; }return 0 ;
builtin/fsck.c +1 -1
@@ -272,7 +272,7 @@ static void check_reachable_object(struct object *obj)if (!(obj->flags &HAS_OBJ )) {if (is_promisor_object (&obj->oid))return ; -if (has_object_pack (&obj->oid)) +if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */ printf_ln(_("missing %s %s"), printable_type(&obj->oid, obj->type),
builtin/pack-objects.c +2 -2
@@ -1529,7 +1529,7 @@ static int want_found_object(const struct object_id *oid, int exclude,return 0 ;if (ignore_packed_keep_in_core && p->pack_keep_in_core)return 0 ; -if (has_object_kept_pack (oid, flags)) +if (has_object_kept_pack (p->repo, oid, flags))return 0 ; }
@@ -3627,7 +3627,7 @@ static void show_cruft_commit(struct commit *commit, void *data) static int cruft_include_check_obj(struct object *obj, void *data UNUSED) { -return !has_object_kept_pack (&obj->oid,IN_CORE_KEEP_PACKS ); +return !has_object_kept_pack (to_pack.repo, &obj->oid,IN_CORE_KEEP_PACKS ); }static int cruft_include_check (struct commit *commit,void *data)
diff.c +2 -1
@@ -4041,7 +4041,8 @@ static int reuse_worktree_file(struct index_state *istate,* objects however would tend to be slower as they need * to be individually opened and inflated. */ -if (!FAST_WORKING_DIRECTORY && !want_file &&has_object_pack (oid)) +if (!FAST_WORKING_DIRECTORY && !want_file && +has_object_pack (istate->repo, oid))return 0 ;/*
list-objects.c +2 -1
@@ -41,7 +41,8 @@ static void show_object(struct traversal_context *ctx, {if (!ctx->show_object)return ; -if (ctx->revs->unpacked &&has_object_pack (&object->oid)) +if (ctx->revs->unpacked &&has_object_pack (ctx->revs->repo, + &object->oid))return ; ctx->show_object (object, name, ctx->show_data);
pack-bitmap.c +1 -1
@@ -1889,7 +1889,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,bitmap_unset (result, i);for (i =0 ; i < eindex->count; ++i) { -if (has_object_pack (&eindex->objects[i]->oid)) +if (has_object_pack (the_repository, &eindex->objects[i]->oid))bitmap_unset (result, objects_nr + i); } }
packfile.c +5 -4
@@ -2143,16 +2143,17 @@ int find_kept_pack_entry(struct repository *r,return 0 ; } -int has_object_pack (const struct object_id *oid) +int has_object_pack (struct repository *r,const struct object_id *oid) {struct pack_entry e; -return find_pack_entry (the_repository, oid, &e); +return find_pack_entry (r, oid, &e); } -int has_object_kept_pack (const struct object_id *oid,unsigned flags) +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags) {struct pack_entry e; -return find_kept_pack_entry (the_repository, oid, flags, &e); +return find_kept_pack_entry (r, oid, flags, &e); } int for_each_object_in_pack(struct packed_git *p,
packfile.h +3 -2
@@ -193,8 +193,9 @@ const struct packed_git *has_packed_and_bad(struct repository *, const struct obint find_pack_entry (struct repository *r,const struct object_id *oid,struct pack_entry *e);int find_kept_pack_entry (struct repository *r,const struct object_id *oid,unsigned flags,struct pack_entry *e); -int has_object_pack (const struct object_id *oid); -int has_object_kept_pack (const struct object_id *oid,unsigned flags); +int has_object_pack (struct repository *r,const struct object_id *oid); +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags);/* * Return 1 if an object in a promisor packfile is or refers to the given
prune-packed.c +1 -1
@@ -24,7 +24,7 @@ static int prune_object(const struct object_id *oid, const char *path, {int *opts = data; -if (!has_object_pack (oid)) +if (!has_object_pack (the_repository, oid))return 0 ;if (*opts &PRUNE_PACKED_DRY_RUN )
reachable.c +1 -1
@@ -239,7 +239,7 @@ static int want_recent_object(struct recent_data *data, const struct object_id *oid) {if (data->ignore_in_core_kept_packs && -has_object_kept_pack (oid,IN_CORE_KEEP_PACKS )) +has_object_kept_pack (data->revs->repo, oid,IN_CORE_KEEP_PACKS ))return 0 ;return 1 ; }
revision.c +2 -2
@@ -4103,10 +4103,10 @@ enum commit_action get_commit_action(struct rev_info *revs, struct commit *commi {if (commit->object.flags &SHOWN )return commit_ignore; -if (revs->unpacked &&has_object_pack (&commit->object.oid)) +if (revs->unpacked &&has_object_pack (revs->repo, &commit->object.oid))return commit_ignore;if (revs->no_kept_objects) { -if (has_object_kept_pack (&commit->object.oid, +if (has_object_kept_pack (revs->repo, &commit->object.oid, revs->keep_pack_cache_flags))return commit_ignore; }
Signature · 1 line
2.47.0
PATCH v8 6/10 packfile: pass down repository to `for_each_packed_object`
In reply to Karthik NayakThe function for_each_packed_object currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from this function and closely related function is_promisor_object.
--- builtin/cat-file.c | 7 ++++--- builtin/fsck.c | 18 +++++++++++------- builtin/pack-objects.c | 7 +++++-- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 2 +- fsck.c | 2 +- list-objects.c | 4 ++-- object-store-ll.h | 4 ++-- packfile.c | 14 +++++++------- packfile.h | 2 +- promisor-remote.c | 2 +- reachable.c | 2 +- revision.c | 9 +++++---- tag.c | 2 +- 15 files changed, 44 insertions(+), 35 deletions(-)
builtin/cat-file.c +4 -3
@@ -827,15 +827,16 @@ static int batch_objects(struct batch_options *opt) cb.seen = &seen;for_each_loose_object (batch_unordered_loose, &cb,0 ); -for_each_packed_object (batch_unordered_packed, &cb, -FOR_EACH_OBJECT_PACK_ORDER ); +for_each_packed_object (the_repository, batch_unordered_packed, + &cb,FOR_EACH_OBJECT_PACK_ORDER );oidset_clear (&seen); }else {struct oid_array sa =OID_ARRAY_INIT ;for_each_loose_object (collect_loose_object, &sa,0 ); -for_each_packed_object (collect_packed_object, &sa,0 ); +for_each_packed_object (the_repository, collect_packed_object, + &sa,0 );oid_array_for_each_unique (&sa, batch_object_cb, &cb);
builtin/fsck.c +11 -7
@@ -150,7 +150,7 @@ static int mark_object(struct object *obj, enum object_type type,return 0 ; obj->flags |=REACHABLE ; -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))/* * Further recursion does not need to be performed on this * object since it is a promisor object (so it does not need to
@@ -270,7 +270,7 @@ static void check_reachable_object(struct object *obj)* do a full fsck */ if (!(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */
@@ -391,7 +391,10 @@ static void check_connectivity(void)* traversal. */ for_each_loose_object (mark_loose_unreachable_referents,NULL ,0 ); -for_each_packed_object (mark_packed_unreachable_referents,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_unreachable_referents, +NULL , +0 ); }/* Look up all the requirements, warn about missing objects.. */
@@ -488,7 +491,7 @@ static void fsck_handle_reflog_oid(const char *refname, struct object_id *oid, refname, timestamp); obj->flags |=USED ;mark_object_reachable (obj); - }else if (!is_promisor_object (oid)) { + }else if (!is_promisor_object (the_repository, oid)) {error (_ ("%s: invalid reflog entry %s" ), refname,oid_to_hex (oid)); errors_found |=ERROR_REACHABLE ;
@@ -531,7 +534,7 @@ static int fsck_handle_ref(const char *refname, const char *referent UNUSED, con obj =parse_object (the_repository, oid);if (!obj) { -if (is_promisor_object (oid)) { +if (is_promisor_object (the_repository, oid)) {/* * Increment default_refs anyway, because this is a * valid ref.
@@ -966,7 +969,8 @@ int cmd_fsck(int argc,if (connectivity_only) {for_each_loose_object (mark_loose_for_connectivity,NULL ,0 ); -for_each_packed_object (mark_packed_for_connectivity,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_for_connectivity,NULL ,0 ); }else {prepare_alt_odb (the_repository);for (odb = the_repository->objects->odb; odb; odb = odb->next)
@@ -1011,7 +1015,7 @@ int cmd_fsck(int argc, &oid);if (!obj || !(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&oid)) +if (is_promisor_object (the_repository, &oid))continue ;error (_ ("%s: object missing" ),oid_to_hex (&oid)); errors_found |=ERROR_OBJECT ;
builtin/pack-objects.c +5 -2
@@ -3858,7 +3858,8 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,* Quietly ignore EXPECTED missing objects. This avoids problems with * staging them now and getting an odd error later. */ -if (!has_object (the_repository, &obj->oid,0 ) &&is_promisor_object (&obj->oid)) +if (!has_object (the_repository, &obj->oid,0 ) && +is_promisor_object (to_pack.repo, &obj->oid))return ;show_object (obj, name, data);
@@ -3927,7 +3928,9 @@ static int add_object_in_unpacked_pack(const struct object_id *oid,static void add_objects_in_unpacked_packs (void ) { - if (for_each_packed_object(add_object_in_unpacked_pack, NULL, + if (for_each_packed_object(to_pack.repo, + add_object_in_unpacked_pack, + NULL, FOR_EACH_OBJECT_PACK_ORDER | FOR_EACH_OBJECT_LOCAL_ONLY | FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS |
builtin/repack.c +1 -1
@@ -404,7 +404,7 @@ static void repack_promisor_objects(const struct pack_objects_args *args,* {type -> existing pack order} ordering when computing deltas instead * of a {type -> size} ordering, which may produce better deltas. */ -for_each_packed_object (write_oid, &cmd, +for_each_packed_object (the_repository, write_oid, &cmd,FOR_EACH_OBJECT_PROMISOR_ONLY );if (cmd.in ==-1 ) {
builtin/rev-list.c +1 -1
@@ -121,7 +121,7 @@ static inline void finish_object__ma(struct object *obj)return ;case MA_ALLOW_PROMISOR : -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;die ("unexpected missing %s object '%s'" ,type_name (obj->type),oid_to_hex (&obj->oid));
commit-graph.c +1 -1
@@ -1960,7 +1960,7 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx) ctx->progress =start_delayed_progress (_ ("Finding commits for commit graph among packed objects" ), ctx->approx_nr_objects); -for_each_packed_object (add_packed_commits, ctx, +for_each_packed_object (ctx->r, add_packed_commits, ctx,FOR_EACH_OBJECT_PACK_ORDER );if (ctx->progress_done < ctx->approx_nr_objects)display_progress (ctx->progress, ctx->approx_nr_objects);
fsck.c +1 -1
@@ -1295,7 +1295,7 @@ static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done, buf =repo_read_object_file (the_repository, oid, &type, &size);if (!buf) { -if (is_promisor_object (oid)) +if (is_promisor_object (the_repository, oid))continue ; ret |= report(options, oid, OBJ_BLOB, msg_missing,
list-objects.c +2 -2
@@ -75,7 +75,7 @@ static void process_blob(struct traversal_context *ctx,*/ if (ctx->revs->exclude_promisor_objects && !repo_has_object_file (the_repository, &obj->oid) && -is_promisor_object (&obj->oid)) +is_promisor_object (ctx->revs->repo, &obj->oid))return ; pathlen = path->len;
@@ -180,7 +180,7 @@ static void process_tree(struct traversal_context *ctx,* an incomplete list of missing objects. */ if (revs->exclude_promisor_objects && -is_promisor_object (&obj->oid)) +is_promisor_object (revs->repo, &obj->oid))return ;if (!revs->do_not_die_on_missing_objects)
object-store-ll.h +2 -2
@@ -550,7 +550,7 @@ typedef int each_packed_object_fn(const struct object_id *oid,int for_each_object_in_pack (struct packed_git *p,each_packed_object_fn ,void *data,enum for_each_object_flags flags); -int for_each_packed_object (each_packed_object_fn ,void *, -enum for_each_object_flags flags); +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags);#endif /* OBJECT_STORE_LL_H */
packfile.c +7 -7
@@ -2200,15 +2200,15 @@ int for_each_object_in_pack(struct packed_git *p,return r; } -int for_each_packed_object (each_packed_object_fn cb,void *data, -enum for_each_object_flags flags) +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags) {struct packed_git *p;int r =0 ;int pack_errors =0 ; -prepare_packed_git (the_repository); -for (p =get_all_packs (the_repository); p; p = p->next) { +prepare_packed_git (repo); +for (p =get_all_packs (repo); p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ; if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
@@ -2286,14 +2286,14 @@ static int add_promisor_object(const struct object_id *oid,return 0 ; } -int is_promisor_object (const struct object_id *oid) +int is_promisor_object (struct repository *r,const struct object_id *oid) {static struct oidset promisor_objects;static int promisor_objects_prepared;if (!promisor_objects_prepared) { -if (repo_has_promisor_remote (the_repository)) { -for_each_packed_object (add_promisor_object, +if (repo_has_promisor_remote (r)) { +for_each_packed_object (r, add_promisor_object, &promisor_objects,FOR_EACH_OBJECT_PROMISOR_ONLY |FOR_EACH_OBJECT_PACK_ORDER );
packfile.h +1 -1
@@ -201,7 +201,7 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,* Return 1 if an object in a promisor packfile is or refers to the given * object, 0 otherwise. */ -int is_promisor_object (const struct object_id *oid); +int is_promisor_object (struct repository *r,const struct object_id *oid);/* * Expose a function for fuzz testing.
promisor-remote.c +1 -1
@@ -283,7 +283,7 @@ void promisor_remote_get_direct(struct repository *repo, }for (i =0 ; i < remaining_nr; i++) { -if (is_promisor_object (&remaining_oids[i])) +if (is_promisor_object (repo, &remaining_oids[i]))die (_ ("could not fetch %s from promisor remote" ),oid_to_hex (&remaining_oids[i])); }
reachable.c +1 -1
@@ -324,7 +324,7 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,if (ignore_in_core_kept_packs) flags |=FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS ; - r =for_each_packed_object (add_recent_packed, &data, flags); + r =for_each_packed_object (revs->repo, add_recent_packed, &data, flags); done:oidset_clear (&data.extra_recent_oids);
revision.c +5 -4
@@ -390,7 +390,8 @@ static struct object *get_reference(struct rev_info *revs, const char *name,if (!object) {if (revs->ignore_missing)return NULL ; -if (revs->exclude_promisor_objects &&is_promisor_object (oid)) +if (revs->exclude_promisor_objects && +is_promisor_object (revs->repo, oid))return NULL ;if (revs->do_not_die_on_missing_objects) {oidset_insert (&revs->missing_commits, oid);
@@ -432,7 +433,7 @@ static struct commit *handle_commit(struct rev_info *revs,if (revs->ignore_missing_links || (flags &UNINTERESTING ))return NULL ;if (revs->exclude_promisor_objects && -is_promisor_object (&tag->tagged->oid)) +is_promisor_object (revs->repo, &tag->tagged->oid))return NULL ;if (revs->do_not_die_on_missing_objects && oid) {oidset_insert (&revs->missing_commits, oid);
@@ -1211,7 +1212,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit, revs->do_not_die_on_missing_objects;if (repo_parse_commit_gently (revs->repo, p, gently) <0 ) {if (revs->exclude_promisor_objects && -is_promisor_object (&p->object.oid)) { +is_promisor_object (revs->repo, &p->object.oid)) {if (revs->first_parent_only)break ;continue ;
@@ -3915,7 +3916,7 @@ int prepare_revision_walk(struct rev_info *revs) revs->treesame.name ="treesame" ;if (revs->exclude_promisor_objects) { -for_each_packed_object (mark_uninteresting, revs, +for_each_packed_object (revs->repo, mark_uninteresting, revs,FOR_EACH_OBJECT_PROMISOR_ONLY ); }
tag.c +1 -1
@@ -84,7 +84,7 @@ struct object *deref_tag(struct repository *r, struct object *o, const char *war o =NULL ; }if (!o && warn) { -if (last_oid &&is_promisor_object (last_oid)) +if (last_oid &&is_promisor_object (r, last_oid))return NULL ;if (!warnlen) warnlen =strlen (warn);
Signature · 1 line
2.47.0
PATCH v8 7/10 config: make `delta_base_cache_limit` a non-global variable
In reply to Karthik NayakThe delta_base_cache_limit variable is a global config variable used by multiple subsystems. Let's make this non-global, by adding this variable independently to the subsystems where it is used.
First, add the setting to the repo_settings struct, this provides access to the config in places where the repository is available. Use this in packfile.c.
In index-pack.c we add it to the pack_idx_option struct and its constructor. While the repository struct is available here, it may not be set because git index-pack can be used without a repository.
In gc.c add it to the gc_config struct and also the constructor function. The gc functions currently do not have direct access to a repository struct.
These changes are made to remove the usage of delta_base_cache_limit as a global variable in packfile.c. This brings us one step closer to removing the USE_THE_REPOSITORY_VARIABLE definition in packfile.c which we complete in the next patch.
--- builtin/gc.c | 8 +++++++- builtin/index-pack.c | 10 +++++++--- config.c | 5 ----- environment.c | 1 - environment.h | 1 - pack-objects.h | 3 ++- pack-write.c | 1 + pack.h | 2 ++ packfile.c | 10 ++++++++-- repo-settings.c | 5 +++++ repo-settings.h | 3 +++ 11 files changed, 35 insertions(+), 14 deletions(-)
builtin/gc.c +7 -1
@@ -138,6 +138,7 @@ struct gc_config {char *repack_filter_to;unsigned long big_pack_threshold;unsigned long max_delta_cache_size; +size_t delta_base_cache_limit; };#define GC_CONFIG_INIT { \
@@ -153,6 +154,7 @@ struct gc_config { .prune_expire = xstrdup("2.weeks.ago"), \ .prune_worktrees_expire = xstrdup("3.months.ago"), \ .max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \ + .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \ }static void gc_config_release (struct gc_config *cfg)
@@ -168,6 +170,7 @@ static void gc_config(struct gc_config *cfg) {const char *value;char *owned =NULL ; +unsigned long ulongval;if (!git_config_get_value ("gc.packrefs" , &value)) {if (value && !strcmp (value,"notbare" ))
@@ -206,6 +209,9 @@ static void gc_config(struct gc_config *cfg)git_config_get_ulong ("gc.bigpackthreshold" , &cfg->big_pack_threshold);git_config_get_ulong ("pack.deltacachesize" , &cfg->max_delta_cache_size); +if (!git_config_get_ulong ("core.deltabasecachelimit" , &ulongval)) + cfg->delta_base_cache_limit = ulongval; +if (!git_config_get_string ("gc.repackfilter" , &owned)) {free (cfg->repack_filter); cfg->repack_filter = owned;
@@ -416,7 +422,7 @@ static uint64_t estimate_repack_memory(struct gc_config *cfg,* read_sha1_file() (either at delta calculation phase, or * writing phase) also fills up the delta base cache */ - heap += delta_base_cache_limit; + heap += cfg->delta_base_cache_limit;/* and of course pack-objects has its own delta cache */ heap += cfg->max_delta_cache_size;
builtin/index-pack.c +7 -3
@@ -1238,7 +1238,7 @@ static void parse_pack_objects(unsigned char *hash)* recursively checking if the resulting object is used as a base * for some more deltas. */ -static void resolve_deltas (void ) +static void resolve_deltas (struct pack_idx_option *opts) {int i;
@@ -1254,7 +1254,7 @@ static void resolve_deltas(void) nr_ref_deltas + nr_ofs_deltas); nr_dispatched =0 ; - base_cache_limit = delta_base_cache_limit * nr_threads; + base_cache_limit = opts->delta_base_cache_limit * nr_threads;if (nr_threads >1 ||getenv ("GIT_FORCE_THREADS" )) {init_thread ();work_lock ();
@@ -1604,6 +1604,10 @@ static int git_index_pack_config(const char *k, const char *v,else opts->flags &= ~WRITE_REV ; } +if (!strcmp (k,"core.deltabasecachelimit" )) { + opts->delta_base_cache_limit =git_config_ulong (k, v, ctx->kvi); +return 0 ; + }return git_default_config (k, v, ctx, cb); }
@@ -1930,7 +1934,7 @@ int cmd_index_pack(int argc,parse_pack_objects (pack_hash);if (report_end_of_input)write_in_full (2 ,"\0" ,1 ); -resolve_deltas (); +resolve_deltas (&opts);conclude_pack (fix_thin_pack, curr_pack, pack_hash);free (ofs_deltas);free (ref_deltas);
config.c +0 -5
@@ -1515,11 +1515,6 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.deltabasecachelimit" )) { - delta_base_cache_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -1
@@ -51,7 +51,6 @@ enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ;size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ;size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ; -size_t delta_base_cache_limit =96 *1024 *1024 ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
environment.h +0 -1
@@ -165,7 +165,6 @@ extern int zlib_compression_level;extern int pack_compression_level;extern size_t packed_git_window_size;extern size_t packed_git_limit; -extern size_t delta_base_cache_limit;extern unsigned long big_file_threshold;extern unsigned long pack_size_limit_cfg;extern int max_allowed_tree_depth;
pack-objects.h +2 -1
@@ -7,7 +7,8 @@struct repository ; -#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_BASE_CACHE_LIMIT (96 * 1024 * 1024)#define OE_DFS_STATE_BITS 2#define OE_DEPTH_BITS 12
pack-write.c +1 -0
@@ -21,6 +21,7 @@ void reset_pack_idx_option(struct pack_idx_option *opts)memset (opts,0 ,sizeof (*opts)); opts->version =2 ; opts->off32_limit =0x7fffffff ; + opts->delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; }static int sha1_compare (const void *_a,const void *_b)
pack.h +2 -0
@@ -58,6 +58,8 @@ struct pack_idx_option {*/ int anomaly_alloc, anomaly_nr;uint32_t *anomaly; + +size_t delta_base_cache_limit; };void reset_pack_idx_option (struct pack_idx_option *);
packfile.c +8 -2
@@ -1496,7 +1496,9 @@ void clear_delta_base_cache(void) }static void add_delta_base_cache (struct packed_git *p,off_t base_offset, -void *base,unsigned long base_size,enum object_type type) +void *base,unsigned long base_size, +unsigned long delta_base_cache_limit, +enum object_type type) {struct delta_base_cache_entry *ent;struct list_head *lru, *tmp;
@@ -1698,6 +1700,8 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,int delta_stack_nr =0 , delta_stack_alloc =UNPACK_ENTRY_STACK_PREALLOC ;int base_from_cache =0 ; +prepare_repo_settings (p->repo); +write_pack_access_log (p, obj_offset);/* PHASE 1: drill down to the innermost base object */
@@ -1878,7 +1882,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,* before we are done using it. */ if (!external_base) -add_delta_base_cache (p, base_obj_offset, base, base_size, type); +add_delta_base_cache (p, base_obj_offset, base, base_size, + p->repo->settings.delta_base_cache_limit, + type);free (delta_data);free (external_base);
repo-settings.c +5 -0
@@ -3,6 +3,7 @@#include "repo-settings.h" #include "repository.h" #include "midx.h" +#include "pack-objects.h" static void repo_cfg_bool (struct repository *r,const char *key,int *dest,int def)
@@ -26,6 +27,7 @@ void prepare_repo_settings(struct repository *r)const char *strval;int manyfiles;int read_changed_paths; +unsigned long ulongval;if (!r->gitdir)BUG ("Cannot add settings for uninitialized repository" );
@@ -123,6 +125,9 @@ void prepare_repo_settings(struct repository *r)* removed. */ r->settings.command_requires_full_index =1 ; + +if (!repo_config_get_ulong (r,"core.deltabasecachelimit" , &ulongval)) + r->settings.delta_base_cache_limit = ulongval; }enum log_refs_config repo_settings_get_log_all_ref_updates (struct repository *repo)
repo-settings.h +3 -0
@@ -57,12 +57,15 @@ struct repo_settings {int core_multi_pack_index;int warn_ambiguous_refs;/* lazily loaded via accessor */ + +size_t delta_base_cache_limit; };#define REPO_SETTINGS_INIT { \ .index_version = -1, \ .core_untracked_cache = UNTRACKED_CACHE_KEEP, \ .fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \ .warn_ambiguous_refs = -1, \ + .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \ }void prepare_repo_settings (struct repository *r);
Signature · 1 line
2.47.0
Re: [PATCH v8 07/10] config: make `delta_base_cache_limit` a non-global variable
Quoted 3 lines · Karthik Nayak
In
gc.cadd it to thegc_configstruct and also the constructor function. The gc functions currently do not have direct access to a repository struct.
We should remember to remove this member from gc_config when we pass the repository through the callchain. As a built-in, cmd_gc() should already be receiving the repository as its parameter, so it may not have to wait for a long time before we are ready to do so.
Perhaps have a comment next to the member's definition to remind us?
Quoted 4 lines
These changes are made to remove the usage of
delta_base_cache_limitas a global variable inpackfile.c. This brings us one step closer to removing theUSE_THE_REPOSITORY_VARIABLEdefinition inpackfile.cwhich we complete in the next patch.
OK.
IIUC, Taylor's comment to avoid repeated calls to configuration layer to obtain delta_base_cache_limit, aside from typofixes, was the only thing remaining for this topic? If so, everything is looking really good.
Re: [PATCH v8 07/10] config: make `delta_base_cache_limit` a non-global variable
Quoted 6 lines · Junio C Hamano
Quoted 3 lines · Karthik Nayak
In
gc.cadd it to thegc_configstruct and also the constructor function. The gc functions currently do not have direct access to a repository struct.
Quoted 7 lines
We should remember to remove this member from
gc_configwhen we pass the repository through the callchain. As a built-in,cmd_gc()should already be receiving the repository as its parameter, so it may not have to wait for a long time before we are ready to do so.Perhaps have a comment next to the member's definition to remind us?
Yeah, that seems like a good idea.
Quoted 11 lines
Quoted 4 lines
These changes are made to remove the usage of
delta_base_cache_limitas a global variable inpackfile.c. This brings us one step closer to removing theUSE_THE_REPOSITORY_VARIABLEdefinition inpackfile.cwhich we complete in the next patch.OK.
IIUC, Taylor's comment to avoid repeated calls to configuration layer to obtain
delta_base_cache_limit, aside from typofixes, was the only thing remaining for this topic? If so, everything is looking really good.
Thanks, will send a new version with the comment inlined.
PATCH v8 8/10 config: make `packed_git_(limit|window_size)` non-global variables
In reply to Karthik NayakThe variables packed_git_window_size and packed_git_limit are global config variables used in the packfile.c file. Since it is only used in this file, let's change it from being a global config variable to a local variable for the subsystem.
With this, we rid packfile.c from all global variable usage and this means we can also remove the USE_THE_REPOSITORY_VARIABLE guard from the file.
--- builtin/fast-import.c | 4 ++-- config.c | 17 ----------------- environment.c | 2 -- packfile.c | 23 +++++++++++++++-------- packfile.h | 2 +- repo-settings.c | 13 +++++++++++++ repo-settings.h | 4 ++++ 7 files changed, 35 insertions(+), 30 deletions(-)
builtin/fast-import.c +2 -2
@@ -3539,7 +3539,7 @@ static void parse_argv(void)int cmd_fast_import (int argc,const char **argv,const char *prefix, - struct repository *repo UNUSED) +struct repository *repo) {unsigned int i;
@@ -3660,7 +3660,7 @@ int cmd_fast_import(int argc,fprintf (stderr," pools: %10lu KiB\n" , (unsigned long )((tree_entry_allocd + fi_mem_pool.pool_alloc) /1024 ));fprintf (stderr," objects: %10" PRIuMAX" KiB\n" , (alloc_count*sizeof (struct object_entry ))/1024 );fprintf (stderr,"---------------------------------------------------------------------\n" ); -pack_report (); +pack_report (repo);fprintf (stderr,"---------------------------------------------------------------------\n" );fprintf (stderr,"\n" ); }
config.c +0 -17
@@ -1493,28 +1493,11 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.packedgitwindowsize" )) { -int pgsz_x2 =getpagesize () *2 ; - packed_git_window_size =git_config_ulong (var, value, ctx->kvi); - -/* This value must be multiple of (pagesize * 2) */ - packed_git_window_size /= pgsz_x2; -if (packed_git_window_size <1 ) - packed_git_window_size =1 ; - packed_git_window_size *= pgsz_x2; -return 0 ; - } -if (!strcmp (var,"core.bigfilethreshold" )) { big_file_threshold =git_config_ulong (var, value, ctx->kvi);return 0 ; } -if (!strcmp (var,"core.packedgitlimit" )) { - packed_git_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -2
@@ -49,8 +49,6 @@ int fsync_object_files = -1;int use_fsync =-1 ;enum fsync_method fsync_method =FSYNC_METHOD_DEFAULT ;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ; -size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ; -size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
packfile.c +15 -8
@@ -1,4 +1,3 @@ -#define USE_THE_REPOSITORY_VARIABLE #include "git-compat-util.h" #include "environment.h"
@@ -46,15 +45,15 @@ static size_t pack_mapped;#define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; } -void pack_report (void ) +void pack_report (struct repository *repo) {fprintf (stderr,"pack_report: getpagesize() = %10" SZ_FMT "\n" "pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" "pack_report: core.packedGitLimit = %10" SZ_FMT "\n" ,sz_fmt (getpagesize ()), -sz_fmt (packed_git_window_size), -sz_fmt (packed_git_limit)); +sz_fmt (repo->settings.packed_git_window_size), +sz_fmt (repo->settings.packed_git_limit));fprintf (stderr,"pack_report: pack_used_ctr = %10u\n" "pack_report: pack_mmap_calls = %10u\n"
@@ -650,8 +649,15 @@ unsigned char *use_pack(struct packed_git *p,break ; }if (!win) { -size_t window_align = packed_git_window_size /2 ; +size_t window_align;off_t len; +struct repo_settings *settings; + +/* lazy load the settings in case it hasn't been setup */ +prepare_repo_settings (p->repo); + settings = &p->repo->settings; + + window_align = settings->packed_git_window_size /2 ;if (p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name);
@@ -659,11 +665,12 @@ unsigned char *use_pack(struct packed_git *p,CALLOC_ARRAY (win,1 ); win->offset = (offset / window_align) * window_align; len = p->pack_size - win->offset; -if (len > packed_git_window_size) - len = packed_git_window_size; +if (len > settings->packed_git_window_size) + len = settings->packed_git_window_size; win->len = (size_t )len; pack_mapped += win->len; -while (packed_git_limit < pack_mapped + +while (settings->packed_git_limit < pack_mapped &&unuse_one_window (p)) ;/* nothing */ win->base = xmmap_gently(NULL, win->len,
packfile.h +1 -1
@@ -89,7 +89,7 @@ unsigned long repo_approximate_object_count(struct repository *r);struct packed_git *find_oid_pack (const struct object_id *oid,struct packed_git *packs); -void pack_report (void ); +void pack_report (struct repository *repo);/* * mmap the index file for the specified packfile (if it is not
repo-settings.c +13 -0
@@ -128,6 +128,19 @@ void prepare_repo_settings(struct repository *r)if (!repo_config_get_ulong (r,"core.deltabasecachelimit" , &ulongval)) r->settings.delta_base_cache_limit = ulongval; + +if (!repo_config_get_ulong (r,"core.packedgitwindowsize" , &ulongval)) { +int pgsz_x2 =getpagesize () *2 ; + +/* This value must be multiple of (pagesize * 2) */ + ulongval /= pgsz_x2; +if (ulongval <1 ) + ulongval =1 ; + r->settings.packed_git_window_size = ulongval * pgsz_x2; + } + +if (!repo_config_get_ulong (r,"core.packedgitlimit" , &ulongval)) + r->settings.packed_git_limit = ulongval; }enum log_refs_config repo_settings_get_log_all_ref_updates (struct repository *repo)
repo-settings.h +4 -0
@@ -59,6 +59,8 @@ struct repo_settings {int warn_ambiguous_refs;/* lazily loaded via accessor */ size_t delta_base_cache_limit; +size_t packed_git_window_size; +size_t packed_git_limit; };#define REPO_SETTINGS_INIT { \ .index_version = -1, \
@@ -66,6 +68,8 @@ struct repo_settings { .fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \ .warn_ambiguous_refs = -1, \ .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \ + .packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE, \ + .packed_git_limit = DEFAULT_PACKED_GIT_LIMIT, \ }void prepare_repo_settings (struct repository *r);
Signature · 1 line
2.47.0
PATCH v8 9/10 midx: add repository to `multi_pack_index` struct
In reply to Karthik NayakThe multi_pack_index struct represents the MIDX for a repository. Here, we add a pointer to the repository in this struct, allowing direct use of the repository variable without relying on the global the_repository struct.
With this addition, we can determine the repository associated with a bitmap_index struct. A bitmap_index points to either a packed_git or a multi_pack_index, both of which have direct repository references. To support this, we introduce a static helper function, bitmap_repo, in pack-bitmap.c, which retrieves a repository given a bitmap_index.
With this, we clear up all usages of the_repository within pack-bitmap.c and also remove the USE_THE_REPOSITORY_VARIABLE definition. Bringing us another step closer to remove all global variable usage.
Although this change also opens up the potential to clean up midx.c, doing so would require additional refactoring to pass the repository struct to functions where the MIDX struct is created: a task better suited for future patches.
--- midx.c | 1 + midx.h | 3 ++ pack-bitmap.c | 90 +++++++++++++++++++++++++++++++-------------------- 3 files changed, 59 insertions(+), 35 deletions(-)
midx.c +1 -0
@@ -131,6 +131,7 @@ static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir m->data = midx_map; m->data_len = midx_size; m->local = local; + m->repo = the_repository; m->signature =get_be32 (m->data);if (m->signature !=MIDX_SIGNATURE )
midx.h +3 -0
@@ -71,6 +71,9 @@ struct multi_pack_index {const char **pack_names;struct packed_git **packs; + +struct repository *repo; +char object_dir[FLEX_ARRAY ]; };
pack-bitmap.c +55 -35
251 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH v8 10/10 packfile.c: remove unnecessary prepare_packed_git() call
In reply to Karthik NayakFrom: Taylor Blau <[email protected]>
In 454ea2e4d7 (treewide: use get_all_packs, 2018-08-20) we converted existing calls to both:
- get_packed_git(), as well as
- the_repository->objects->packed_git
, to instead use the new get_all_packs() function.
In the instance that this commit addresses, there was a preceding call to prepare_packed_git(), which dates all the way back to 660c889e46 (sha1_file: add for_each iterators for loose and packed objects, 2014-10-15) when its caller (for_each_packed_object()) was first introduced.
This call could have been removed in 454ea2e4d7, since get_all_packs() itself calls prepare_packed_git(). But the translation in 454ea2e4d7 was (to the best of my knowledge) a find-and-replace rather than inspecting each individual caller.
Having an extra prepare_packed_git() call here is harmless, since it will notice that we have already set the 'packed_git_initialized' field and the call will be a noop. So we're only talking about a few dozen CPU cycles to set up and tear down the stack frame.
But having a lone prepare_packed_git() call immediately before a call to get_all_packs() confused me, so let's remove it as redundant to avoid more confusion in the future.
--- packfile.c | 1 - 1 file changed, 1 deletion(-)
packfile.c +0 -1
@@ -2220,7 +2220,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,int r =0 ;int pack_errors =0 ; -prepare_packed_git (repo);for (p =get_all_packs (repo); p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ;
Signature · 1 line
2.47.0
v9 cover packfile: avoid using the 'the_repository' global variable
In reply to Karthik NayakThe packfile.c file uses the global variable 'the_repository' extensively throughout the code. Let's remove all usecases of this, by modifying the required functions to accept a 'struct repository' instead. This is to clean up usage of global state.
The first 3 patches are mostly internal to packfile.c, we add the repository field to the packed_git struct and this is used to clear up some useages of the global variables.
The next 3 patches are more disruptive, they modify the function definition of odb_pack_name, has_object[_kept]_pack and for_each_packed_object to receive a repository, helping remove other usages of 'the_repository' variable.
Finally, the next two patches deal with global config values. These values are localized. The last patch is removal of an unecessary call to prepare_packed_git(). For v5 onwards, I've rebased the series off the master: 8f8d6eee53 (The seventh batch, 2024-11-01), as a dependency for this series 'jk/dumb-http-finalize' was merged to master. I've found no conflicts while merging with seen & next. But since this series does touch multiple files, there could be future conflicts.
Changes in v9:
- Added a comment in gc_config to indicate that eventually the delta_base_cache_limit variable should be used through repo_settings.
Changes in v8:
- Fix typos in comments
- For packfile.c use delta_base_cache_limit from the repository settings, this avoids loading the config in hot paths.
- Rename longval to ulongval to better signify the type.
Changes in v7:
- Cleanup stale commit message.
- Add missing space in if statement.
- Fix typo s/incase/in case/.
Changes in v6:
- Lazy load repository settings in packfile.c. This ensures that the settings are available for sure and we do not rely on callees setting it.
- Use size_t for delta_base_cache_limit.
Changes in v5:
- Move packed_git* settings to repo_settings to ensure we don't keep reparsing the settings in use_pack.
Changes in v4:
- Renamed the repository field within packed_git and multi_pack_index from r to repo, while keeping function parameters to be r.
- Fixed bad braces.
Changes in v3:
- Improved commit messages. In the first commit to talk about how packed_git struct could also be part of the alternates of a repository. In the 7th commit to talk about the motive behind removing the global variable.
- Changed 'packed_git->repo' to 'packed_git->r' to keep it consistent with the rest of the code base.
- Replaced 'the_repository' with locally available access to the repository struct in multiple regions.
- Removed unecessary inclusion of the 'repository.h' header file by forward declaring the 'repository' struct.
- Replace memcpy with hashcpy.
- Change the logic in the 7th patch to use if else statements.
- Added an extra commit to cleanup pack-bitmap.c.
Karthik Nayak (9):
packfile: add repository to struct packed_git
packfile: use repository from packed_git directly
packfile: pass repository to static function in the file
packfile: pass down repository to odb_pack_name
packfile: pass down repository to has_object[_kept]_pack
packfile: pass down repository to for_each_packed_object
config: make delta_base_cache_limit a non-global variable
config: make packed_git_(limit|window_size) non-global variables
midx: add repository to multi_pack_index struct
Taylor Blau (1):
packfile.c: remove unnecessary prepare_packed_git() call
builtin/cat-file.c | 7 +- builtin/count-objects.c | 2 +- builtin/fast-import.c | 15 ++-- builtin/fsck.c | 20 +++--- builtin/gc.c | 12 +++- builtin/index-pack.c | 20 ++++-- builtin/pack-objects.c | 11 +-- builtin/pack-redundant.c | 2 +- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 4 +- config.c | 22 ------ connected.c | 3 +- diff.c | 3 +- environment.c | 3 - environment.h | 1 - fsck.c | 2 +- http.c | 4 +- list-objects.c | 7 +- midx-write.c | 2 +- midx.c | 3 +- midx.h | 3 + object-store-ll.h | 9 ++- pack-bitmap.c | 90 ++++++++++++++---------- pack-objects.h | 3 +- pack-write.c | 1 + pack.h | 2 + packfile.c | 144 ++++++++++++++++++++++----------------- packfile.h | 18 +++-- promisor-remote.c | 2 +- prune-packed.c | 2 +- reachable.c | 4 +- repo-settings.c | 18 +++++ repo-settings.h | 7 ++ revision.c | 13 ++-- tag.c | 2 +- 36 files changed, 275 insertions(+), 190 deletions(-)
Range-diff against v8:
-: ---------- > 1: d1fdd6996a packfile: add repository to struct `packed_git`
-: ---------- > 2: 65c09858ce packfile: use `repository` from `packed_git` directly
-: ---------- > 3: 80632934d1 packfile: pass `repository` to static function in the file
-: ---------- > 4: 67d71eab83 packfile: pass down repository to `odb_pack_name`
-: ---------- > 5: ee210fa153 packfile: pass down repository to `has_object[_kept]_pack`
-: ---------- > 6: 8db7094f4e packfile: pass down repository to `for_each_packed_object`
1: d1b6e8801b ! 7: a66494384d config: make `delta_base_cache_limit` a non-global variable
@@ builtin/gc.c: struct gc_config {
char *repack_filter_to;
unsigned long big_pack_threshold;
unsigned long max_delta_cache_size;
++ /*
++ * Remove this member from gc_config once repo_settings is passed
++ * through the callchain.
++ */
+ size_t delta_base_cache_limit;
};2: 30a52f192f = 8: bce9196f6b config: make `packed_git_(limit|window_size)` non-global variables 3: 2fe5d2506f = 9: c7fba8cf6a midx: add repository to `multi_pack_index` struct 4: 05989c2e27 = 10: d7f475fbd0 packfile.c: remove unnecessary prepare_packed_git() call
Signature · 1 line
2.47.0
PATCH v9 1/10 packfile: add repository to struct `packed_git`
The struct packed_git holds information regarding a packed object file. Let's add the repository variable to this object, to represent the repository that this packfile belongs to. This helps remove dependency on the global the_repository object in packfile.c by simply using repository information now readily available in the struct.
We do need to consider that a pack file could be part of the alternates of a repository, but considering that we only have one repository struct and also that we currently anyways use 'the_repository'. We should be OK with this change.
We also modify alloc_packed_git to ensure that the repository is added to newly created packed_git structs. This requires modifying the function and all its callee to pass the repository object down the levels.
--- builtin/fast-import.c | 3 ++- builtin/index-pack.c | 6 ++++-- commit-graph.c | 2 +- connected.c | 3 ++- http.c | 2 +- midx-write.c | 2 +- midx.c | 2 +- object-store-ll.h | 5 +++++ packfile.c | 15 +++++++++------ packfile.h | 6 ++++-- 10 files changed, 30 insertions(+), 16 deletions(-)
builtin/fast-import.c +2 -1
@@ -765,6 +765,7 @@ static void start_packfile(void) p->pack_fd = pack_fd; p->do_not_close =1 ; + p->repo = the_repository; pack_file =hashfd (pack_fd, p->pack_name); pack_data = p;
@@ -888,7 +889,7 @@ static void end_packfile(void) idx_name =keep_pack (create_index ());/* Register the packfile with core git's machinery. */ - new_p =add_packed_git (idx_name,strlen (idx_name),1 ); + new_p =add_packed_git (pack_data->repo, idx_name,strlen (idx_name),1 );if (!new_p)die ("core git rejected index %s" , idx_name); all_packs[pack_id] = new_p;
builtin/index-pack.c +4 -2
@@ -1552,7 +1552,8 @@ static void final(const char *final_pack_name, const char *curr_pack_name,if (do_fsck_object) {struct packed_git *p; - p =add_packed_git (final_index_name,strlen (final_index_name),0 ); + p =add_packed_git (the_repository, final_index_name, +strlen (final_index_name),0 );if (p)install_packed_git (the_repository, p); }
@@ -1650,7 +1651,8 @@ static void read_v2_anomalous_offsets(struct packed_git *p,static void read_idx_option (struct pack_idx_option *opts,const char *pack_name) { -struct packed_git *p =add_packed_git (pack_name,strlen (pack_name),1 ); +struct packed_git *p =add_packed_git (the_repository, pack_name, +strlen (pack_name),1 );if (!p)die (_ ("Cannot open existing pack file '%s'" ), pack_name);
commit-graph.c +1 -1
@@ -1914,7 +1914,7 @@ static int fill_oids_from_packs(struct write_commit_graph_context *ctx,struct packed_git *p;strbuf_setlen (&packname, dirlen);strbuf_addstr (&packname, pack_indexes->items[i].string); - p =add_packed_git (packname.buf, packname.len,1 ); + p =add_packed_git (ctx->r, packname.buf, packname.len,1 );if (!p) { ret =error (_ ("error adding pack %s" ), packname.buf); goto cleanup;
connected.c +2 -1
@@ -54,7 +54,8 @@ int check_connected(oid_iterate_fn fn, void *cb_data,strbuf_add (&idx_file, transport->pack_lockfiles.items[0 ].string, base_len);strbuf_addstr (&idx_file,".idx" ); - new_pack =add_packed_git (idx_file.buf, idx_file.len,1 ); + new_pack =add_packed_git (the_repository, idx_file.buf, + idx_file.len,1 );strbuf_release (&idx_file); }
http.c +1 -1
@@ -2439,7 +2439,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,if (!tmp_idx)return -1 ; - new_pack =parse_pack_index (sha1, tmp_idx); + new_pack =parse_pack_index (the_repository, sha1, tmp_idx);if (!new_pack) {unlink (tmp_idx);free (tmp_idx);
midx-write.c +1 -1
@@ -154,7 +154,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,return ;ALLOC_GROW (ctx->info, ctx->nr +1 , ctx->alloc); - p =add_packed_git (full_path, full_path_len,0 ); + p =add_packed_git (the_repository, full_path, full_path_len,0 );if (!p) {warning (_ ("failed to add packfile '%s'" ), full_path);
midx.c +1 -1
@@ -464,7 +464,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,strhash (key.buf), key.buf, struct packed_git, packmap_ent);if (!p) { - p =add_packed_git (pack_name.buf, pack_name.len, m->local); + p =add_packed_git (r, pack_name.buf, pack_name.len, m->local);if (p) {install_packed_git (r, p);list_add_tail (&p->mru, &r->objects->packed_git_mru);
object-store-ll.h +5 -0
@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;struct object_directory {struct object_directory *next;
@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo denotes the repository this packfile belongs to */ +struct repository *repo; +/* something like ".git/objects/pack/xxxxx.pack" */ char pack_name[FLEX_ARRAY ];/* more */ };
packfile.c +9 -6
@@ -217,11 +217,12 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)return ntohl (level1_ofs[value]); } -static struct packed_git *alloc_packed_git (int extra) +static struct packed_git *alloc_packed_git (struct repository *r,int extra) {struct packed_git *p =xmalloc (st_add (sizeof (*p), extra));memset (p,0 ,sizeof (*p)); p->pack_fd =-1 ; + p->repo = r;return p; }
@@ -233,11 +234,12 @@ static char *pack_path_from_idx(const char *idx_path)return xstrfmt ("%.*s.pack" , (int )len, idx_path); } -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path) {char *path =pack_path_from_idx (idx_path);size_t alloc =st_add (strlen (path),1 ); -struct packed_git *p =alloc_packed_git (alloc); +struct packed_git *p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path);
@@ -703,7 +705,8 @@ void unuse_pack(struct pack_window **w_cursor) } } -struct packed_git *add_packed_git (const char *path,size_t path_len,int local) +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local) {struct stat st;size_t alloc;
@@ -721,7 +724,7 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)* the use xsnprintf double-checks that) */ alloc =st_add3 (path_len,strlen (".promisor" ),1 ); - p =alloc_packed_git (alloc); + p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, path_len);xsnprintf (p->pack_name + path_len, alloc - path_len,".keep" );
@@ -877,7 +880,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,/* Don't reopen a pack we already have. */ if (!hashmap_get (&data->r->objects->pack_map, &hent, pack_name)) { - p =add_packed_git (full_name, full_name_len, data->local); + p =add_packed_git (data->r, full_name, full_name_len, data->local);if (p)install_packed_git (data->r, p); }
packfile.h +4 -2
@@ -46,7 +46,8 @@ const char *pack_basename(struct packed_git *p);* and does not add the resulting packed_git struct to the internal list of * packs. You probably want add_packed_git() instead. */ -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path);typedef void each_file_in_pack_dir_fn (const char *full_path,size_t full_path_len,const char *file_name,void *data);
@@ -113,7 +114,8 @@ void close_pack(struct packed_git *);void close_object_store (struct raw_object_store *o);void unuse_pack (struct pack_window **);void clear_delta_base_cache (void ); -struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local);/* * Unlink the .pack and associated extension files.
Signature · 1 line
2.47.0
Re: [PATCH v9 01/10] packfile: add repository to struct `packed_git`
Quoted 4 lines · Karthik Nayak
We do need to consider that a pack file could be part of the alternates of a repository, but considering that we only have one repository struct and also that we currently anyways use '
the_repository'. We should be OK with this change.
The “but considering” needs for a conclusion to come in the same sentence. But it seems that the conclusion has been put off to the next
sentence: “We should be OK with this change”. That doesn’t read right to me.
Re: [PATCH v9 01/10] packfile: add repository to struct `packed_git`
Quoted 10 lines · Kristoffer Haugsbakk
Quoted 4 lines · Karthik Nayak
We do need to consider that a pack file could be part of the alternates of a repository, but considering that we only have one repository struct and also that we currently anyways use '
the_repository'. We should be OK with this change.The “but considering” needs for a conclusion to come in the same sentence. But it seems that the conclusion has been put off to the next
sentence: “We should be OK with this change”. That doesn’t read right to me.
Hmm, yeah, perhaps a comma right after the 'the_repository' instead of a period would make it better.
PATCH v9 2/10 packfile: use `repository` from `packed_git` directly
In reply to Karthik NayakIn the previous commit, we introduced the repository structure inside packed_git. This provides an alternative route instead of using the global the_repository variable. Let's modify packfile.c now to use this field wherever possible instead of relying on the global state. There are still a few instances of the_repository usage in the file, where there is no struct packed_git locally available, which will be fixed in the following commits.
--- packfile.c | 50 +++++++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 23 deletions(-)
packfile.c +27 -23
@@ -79,7 +79,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)size_t idx_size;int fd =git_open (path), ret;struct stat st; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (fd <0 )return -1 ;
@@ -243,7 +243,7 @@ struct packed_git *parse_pack_index(struct repository *r, unsigned char *sha1,memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path); -hashcpy (p->hash, sha1, the_repository->hash_algo); +hashcpy (p->hash, sha1, p->repo->hash_algo);if (check_packed_git_idx (idx_path, p)) {free (p);return NULL ;
@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = the_repository->objects->packed_git; p; p = p->next) +for (p = current->repo->objects->packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -540,7 +540,7 @@ static int open_packed_git_1(struct packed_git *p)unsigned char hash[GIT_MAX_RAWSZ ];unsigned char *idx_hash;ssize_t read_result; -const unsigned hashsz = the_hash_algo->rawsz; +const unsigned hashsz = p->repo->hash_algo->rawsz;if (open_pack_index (p))return error ("packfile %s index unavailable" , p->pack_name);
@@ -597,7 +597,7 @@ static int open_packed_git_1(struct packed_git *p)if (read_result != hashsz)return error ("packfile %s signature is unavailable" , p->pack_name); idx_hash = ((unsigned char *)p->index_data) + p->index_size - hashsz *2 ; -if (!hasheq (hash, idx_hash, the_repository->hash_algo)) +if (!hasheq (hash, idx_hash, p->repo->hash_algo))return error ("packfile %s does not match index" , p->pack_name);return 0 ; }
@@ -637,7 +637,7 @@ unsigned char *use_pack(struct packed_git *p,*/ if (!p->pack_size && p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name); -if (offset > (p->pack_size - the_hash_algo->rawsz)) +if (offset > (p->pack_size - p->repo->hash_algo->rawsz))die ("offset beyond end of packfile (truncated pack?)" );if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" ));
@@ -711,6 +711,7 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,struct stat st;size_t alloc;struct packed_git *p; +struct object_id oid;/* * Make sure a corresponding .pack file exists and that
@@ -751,9 +752,13 @@ struct packed_git *add_packed_git(struct repository *r, const char *path, p->pack_size = st.st_size; p->pack_local = local; p->mtime = st.st_mtime; -if (path_len < the_hash_algo->hexsz || -get_hash_hex (path + path_len - the_hash_algo->hexsz, p->hash)) -hashclr (p->hash, the_repository->hash_algo); +if (path_len < r->hash_algo->hexsz || +get_oid_hex_algop (path + path_len - r->hash_algo->hexsz, &oid, + r->hash_algo)) +hashclr (p->hash, r->hash_algo); +else +hashcpy (p->hash, oid.hash, r->hash_algo); +return p; }
@@ -1243,9 +1248,9 @@ off_t get_delta_base(struct packed_git *p, }else if (type ==OBJ_REF_DELTA ) {/* The base entry _must_ be in the same pack */ struct object_id oid; -oidread (&oid, base_info, the_repository->hash_algo); +oidread (&oid, base_info, p->repo->hash_algo); base_offset =find_pack_entry_one (&oid, p); - *curpos += the_hash_algo->rawsz; + *curpos += p->repo->hash_algo->rawsz; }else die ("I am totally screwed" );return base_offset;
@@ -1266,7 +1271,7 @@ static int get_delta_base_oid(struct packed_git *p, {if (type ==OBJ_REF_DELTA ) {unsigned char *base =use_pack (p, w_curs, curpos,NULL ); -oidread (oid, base, the_repository->hash_algo); +oidread (oid, base, p->repo->hash_algo);return 0 ; }else if (type ==OBJ_OFS_DELTA ) {uint32_t base_pos;
@@ -1608,7 +1613,7 @@ int packed_object_info(struct repository *r, struct packed_git *p, goto out; } }else -oidclr (oi->delta_base_oid, the_repository->hash_algo); +oidclr (oi->delta_base_oid, p->repo->hash_algo); } oi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :
@@ -1897,7 +1902,7 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32 {const unsigned char *index_fanout = p->index_data;const unsigned char *index_lookup; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;int index_lookup_width;if (!index_fanout)
@@ -1922,7 +1927,7 @@ int nth_packed_object_id(struct object_id *oid, uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (!index) {if (open_pack_index (p))return -1 ;
@@ -1933,11 +1938,10 @@ int nth_packed_object_id(struct object_id *oid, index +=4 *256 ;if (p->index_version ==1 ) {oidread (oid, index +st_add (st_mult (hashsz +4 , n),4 ), - the_repository->hash_algo); + p->repo->hash_algo); }else { index +=8 ; -oidread (oid, index +st_mult (hashsz, n), - the_repository->hash_algo); +oidread (oid, index +st_mult (hashsz, n), p->repo->hash_algo); }return 0 ; }
@@ -1959,7 +1963,7 @@ void check_pack_index_ptr(const struct packed_git *p, const void *vptr)off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz; index +=4 *256 ;if (p->index_version ==1 ) {return ntohl (*((uint32_t *)(index +st_mult (hashsz +4 , n))));
@@ -2159,7 +2163,7 @@ int for_each_object_in_pack(struct packed_git *p,int r =0 ;if (flags &FOR_EACH_OBJECT_PACK_ORDER ) { -if (load_pack_revindex (the_repository, p)) +if (load_pack_revindex (p->repo, p))return -1 ; }
@@ -2227,7 +2231,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, }static int add_promisor_object (const struct object_id *oid, - struct packed_git *pack UNUSED, +struct packed_git *pack, uint32_t pos UNUSED,void *set_) {
@@ -2235,12 +2239,12 @@ static int add_promisor_object(const struct object_id *oid,struct object *obj;int we_parsed_object; - obj =lookup_object (the_repository, oid); + obj =lookup_object (pack->repo, oid);if (obj && obj->parsed) { we_parsed_object =0 ; }else { we_parsed_object =1 ; - obj =parse_object (the_repository, oid); + obj =parse_object (pack->repo, oid); }if (!obj)
Signature · 1 line
2.47.0
PATCH v9 3/10 packfile: pass `repository` to static function in the file
In reply to Karthik NayakSome of the static functions in the packfile.c access global variables, which can simply be avoiding by passing the repository struct down to them. Let's do that.
--- packfile.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
packfile.c +8 -7
@@ -460,13 +460,13 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc *accept_windows_inuse = has_windows_inuse; } -static int close_one_pack (void ) +static int close_one_pack (struct repository *r) {struct packed_git *p, *lru_p =NULL ;struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = the_repository->objects->packed_git; p; p = p->next) { +for (p = r->objects->packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -555,7 +555,7 @@ static int open_packed_git_1(struct packed_git *p) pack_max_fds =1 ; } -while (pack_max_fds <= pack_open_fds &&close_one_pack ()) +while (pack_max_fds <= pack_open_fds &&close_one_pack (p->repo)) ;/* nothing */ p->pack_fd =git_open (p->pack_name);
@@ -610,7 +610,8 @@ static int open_packed_git(struct packed_git *p)return -1 ; } -static int in_window (struct pack_window *win,off_t offset) +static int in_window (struct repository *r,struct pack_window *win, +off_t offset) {/* We must promise at least one full hash after the * offset is available from this window, otherwise the offset
@@ -620,7 +621,7 @@ static int in_window(struct pack_window *win, off_t offset)*/ off_t win_off = win->offset;return win_off <= offset - && (offset + the_hash_algo->rawsz) <= (win_off + win->len); + && (offset + r->hash_algo->rawsz) <= (win_off + win->len); } unsigned char *use_pack(struct packed_git *p,
@@ -642,11 +643,11 @@ unsigned char *use_pack(struct packed_git *p,if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" )); -if (!win || !in_window (win, offset)) { +if (!win || !in_window (p->repo, win, offset)) {if (win) win->inuse_cnt--;for (win = p->windows; win; win = win->next) { -if (in_window (win, offset)) +if (in_window (p->repo, win, offset))break ; }if (!win) {
Signature · 1 line
2.47.0
Re: [PATCH v9 03/10] packfile: pass `repository` to static function in the file
Quoted 2 lines · Karthik Nayak
Some of the static functions in the
packfile.caccess global variables, which can simply be avoiding by passing therepository
s/can simply be avoiding/can simply be avoided/
Re: [PATCH v9 03/10] packfile: pass `repository` to static function in the file
Quoted 5 lines · Kristoffer Haugsbakk
Quoted 2 lines · Karthik Nayak
Some of the static functions in the
packfile.caccess global variables, which can simply be avoiding by passing therepositorys/can simply be avoiding/can simply be avoided/
Indeed, thanks for noticing. Will avoid an immediate re-roll, but have added it locally.
PATCH v9 4/10 packfile: pass down repository to `odb_pack_name`
In reply to Karthik NayakThe function odb_pack_name currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers.
--- builtin/fast-import.c | 8 ++++---- builtin/index-pack.c | 4 ++-- builtin/pack-redundant.c | 2 +- http.c | 2 +- packfile.c | 9 ++++----- packfile.h | 3 ++- 6 files changed, 14 insertions(+), 14 deletions(-)
builtin/fast-import.c +4 -4
@@ -806,7 +806,7 @@ static char *keep_pack(const char *curr_index_name)struct strbuf name =STRBUF_INIT ;int keep_fd; -odb_pack_name (&name, pack_data->hash,"keep" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"keep" ); keep_fd =odb_pack_keep (name.buf);if (keep_fd <0 )die_errno ("cannot create keep file" );
@@ -814,11 +814,11 @@ static char *keep_pack(const char *curr_index_name)if (close (keep_fd))die_errno ("failed to write keep file" ); -odb_pack_name (&name, pack_data->hash,"pack" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"pack" );if (finalize_object_file (pack_data->pack_name, name.buf))die ("cannot store pack file" ); -odb_pack_name (&name, pack_data->hash,"idx" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"idx" );if (finalize_object_file (curr_index_name, name.buf))die ("cannot store index file" );free ((void *)curr_index_name);
@@ -832,7 +832,7 @@ static void unkeep_all_packs(void)for (k =0 ; k < pack_id; k++) {struct packed_git *p = all_packs[k]; -odb_pack_name (&name, p->hash,"keep" ); +odb_pack_name (p->repo, &name, p->hash,"keep" );unlink_or_warn (name.buf); }strbuf_release (&name);
builtin/index-pack.c +2 -2
@@ -1479,7 +1479,7 @@ static void write_special_file(const char *suffix, const char *msg,if (pack_name) filename =derive_filename (pack_name,"pack" , suffix, &name_buf);else - filename =odb_pack_name (&name_buf, hash, suffix); + filename =odb_pack_name (the_repository, &name_buf, hash, suffix); fd =odb_pack_keep (filename);if (fd <0 ) {
@@ -1507,7 +1507,7 @@ static void rename_tmp_packfile(const char **final_name, {if (!*final_name ||strcmp (*final_name, curr_name)) {if (!*final_name) - *final_name =odb_pack_name (name, hash, ext); + *final_name =odb_pack_name (the_repository, name, hash, ext);if (finalize_object_file (curr_name, *final_name))die (_ ("unable to rename temporary '*.%s' file to '%s'" ), ext, *final_name);
builtin/pack-redundant.c +1 -1
@@ -690,7 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix UNUSED, s pl = red =pack_list_difference (local_packs, min);while (pl) {printf ("%s\n%s\n" , -odb_pack_name (&idx_name, pl->pack->hash,"idx" ), +odb_pack_name (pl->pack->repo, &idx_name, pl->pack->hash,"idx" ), pl->pack->pack_name); pl = pl->next; }
http.c +1 -1
@@ -2581,7 +2581,7 @@ struct http_pack_request *new_direct_http_pack_request( preq->url = url; -odb_pack_name (&preq->tmpfile, packed_git_hash,"pack" ); +odb_pack_name (the_repository, &preq->tmpfile, packed_git_hash,"pack" );strbuf_addstr (&preq->tmpfile,".temp" ); preq->packfile =fopen (preq->tmpfile.buf,"a" );if (!preq->packfile) {
packfile.c +4 -5
@@ -25,13 +25,12 @@#include "pack-revindex.h" #include "promisor-remote.h" -char *odb_pack_name (struct strbuf *buf, -const unsigned char *hash, -const char *ext) +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext) {strbuf_reset (buf); -strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (the_repository), -hash_to_hex (hash), ext); +strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (r), +hash_to_hex_algop (hash, r->hash_algo), ext);return buf->buf; }
packfile.h +2 -1
@@ -29,7 +29,8 @@ struct pack_entry {* * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" */ -char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext);/* * Return the basename of the packfile, omitting any containing directory
Signature · 1 line
2.47.0
PATCH v9 5/10 packfile: pass down repository to `has_object[_kept]_pack`
In reply to Karthik NayakThe functions has_object[_kept]_pack currently rely on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from these functions and any related ones.
--- builtin/count-objects.c | 2 +- builtin/fsck.c | 2 +- builtin/pack-objects.c | 4 ++-- diff.c | 3 ++- list-objects.c | 3 ++- pack-bitmap.c | 2 +- packfile.c | 9 +++++---- packfile.h | 5 +++-- prune-packed.c | 2 +- reachable.c | 2 +- revision.c | 4 ++-- 11 files changed, 21 insertions(+), 17 deletions(-)
builtin/count-objects.c +1 -1
@@ -67,7 +67,7 @@ static int count_loose(const struct object_id *oid, const char *path,else { loose_size +=on_disk_bytes (st); loose++; -if (verbose &&has_object_pack (oid)) +if (verbose &&has_object_pack (the_repository, oid)) packed_loose++; }return 0 ;
builtin/fsck.c +1 -1
@@ -272,7 +272,7 @@ static void check_reachable_object(struct object *obj)if (!(obj->flags &HAS_OBJ )) {if (is_promisor_object (&obj->oid))return ; -if (has_object_pack (&obj->oid)) +if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */ printf_ln(_("missing %s %s"), printable_type(&obj->oid, obj->type),
builtin/pack-objects.c +2 -2
@@ -1529,7 +1529,7 @@ static int want_found_object(const struct object_id *oid, int exclude,return 0 ;if (ignore_packed_keep_in_core && p->pack_keep_in_core)return 0 ; -if (has_object_kept_pack (oid, flags)) +if (has_object_kept_pack (p->repo, oid, flags))return 0 ; }
@@ -3627,7 +3627,7 @@ static void show_cruft_commit(struct commit *commit, void *data) static int cruft_include_check_obj(struct object *obj, void *data UNUSED) { -return !has_object_kept_pack (&obj->oid,IN_CORE_KEEP_PACKS ); +return !has_object_kept_pack (to_pack.repo, &obj->oid,IN_CORE_KEEP_PACKS ); }static int cruft_include_check (struct commit *commit,void *data)
diff.c +2 -1
@@ -4041,7 +4041,8 @@ static int reuse_worktree_file(struct index_state *istate,* objects however would tend to be slower as they need * to be individually opened and inflated. */ -if (!FAST_WORKING_DIRECTORY && !want_file &&has_object_pack (oid)) +if (!FAST_WORKING_DIRECTORY && !want_file && +has_object_pack (istate->repo, oid))return 0 ;/*
list-objects.c +2 -1
@@ -41,7 +41,8 @@ static void show_object(struct traversal_context *ctx, {if (!ctx->show_object)return ; -if (ctx->revs->unpacked &&has_object_pack (&object->oid)) +if (ctx->revs->unpacked &&has_object_pack (ctx->revs->repo, + &object->oid))return ; ctx->show_object (object, name, ctx->show_data);
pack-bitmap.c +1 -1
@@ -1889,7 +1889,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,bitmap_unset (result, i);for (i =0 ; i < eindex->count; ++i) { -if (has_object_pack (&eindex->objects[i]->oid)) +if (has_object_pack (the_repository, &eindex->objects[i]->oid))bitmap_unset (result, objects_nr + i); } }
packfile.c +5 -4
@@ -2143,16 +2143,17 @@ int find_kept_pack_entry(struct repository *r,return 0 ; } -int has_object_pack (const struct object_id *oid) +int has_object_pack (struct repository *r,const struct object_id *oid) {struct pack_entry e; -return find_pack_entry (the_repository, oid, &e); +return find_pack_entry (r, oid, &e); } -int has_object_kept_pack (const struct object_id *oid,unsigned flags) +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags) {struct pack_entry e; -return find_kept_pack_entry (the_repository, oid, flags, &e); +return find_kept_pack_entry (r, oid, flags, &e); } int for_each_object_in_pack(struct packed_git *p,
packfile.h +3 -2
@@ -193,8 +193,9 @@ const struct packed_git *has_packed_and_bad(struct repository *, const struct obint find_pack_entry (struct repository *r,const struct object_id *oid,struct pack_entry *e);int find_kept_pack_entry (struct repository *r,const struct object_id *oid,unsigned flags,struct pack_entry *e); -int has_object_pack (const struct object_id *oid); -int has_object_kept_pack (const struct object_id *oid,unsigned flags); +int has_object_pack (struct repository *r,const struct object_id *oid); +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags);/* * Return 1 if an object in a promisor packfile is or refers to the given
prune-packed.c +1 -1
@@ -24,7 +24,7 @@ static int prune_object(const struct object_id *oid, const char *path, {int *opts = data; -if (!has_object_pack (oid)) +if (!has_object_pack (the_repository, oid))return 0 ;if (*opts &PRUNE_PACKED_DRY_RUN )
reachable.c +1 -1
@@ -239,7 +239,7 @@ static int want_recent_object(struct recent_data *data, const struct object_id *oid) {if (data->ignore_in_core_kept_packs && -has_object_kept_pack (oid,IN_CORE_KEEP_PACKS )) +has_object_kept_pack (data->revs->repo, oid,IN_CORE_KEEP_PACKS ))return 0 ;return 1 ; }
revision.c +2 -2
@@ -4103,10 +4103,10 @@ enum commit_action get_commit_action(struct rev_info *revs, struct commit *commi {if (commit->object.flags &SHOWN )return commit_ignore; -if (revs->unpacked &&has_object_pack (&commit->object.oid)) +if (revs->unpacked &&has_object_pack (revs->repo, &commit->object.oid))return commit_ignore;if (revs->no_kept_objects) { -if (has_object_kept_pack (&commit->object.oid, +if (has_object_kept_pack (revs->repo, &commit->object.oid, revs->keep_pack_cache_flags))return commit_ignore; }
Signature · 1 line
2.47.0
PATCH v9 6/10 packfile: pass down repository to `for_each_packed_object`
In reply to Karthik NayakThe function for_each_packed_object currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from this function and closely related function is_promisor_object.
--- builtin/cat-file.c | 7 ++++--- builtin/fsck.c | 18 +++++++++++------- builtin/pack-objects.c | 7 +++++-- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 2 +- fsck.c | 2 +- list-objects.c | 4 ++-- object-store-ll.h | 4 ++-- packfile.c | 14 +++++++------- packfile.h | 2 +- promisor-remote.c | 2 +- reachable.c | 2 +- revision.c | 9 +++++---- tag.c | 2 +- 15 files changed, 44 insertions(+), 35 deletions(-)
builtin/cat-file.c +4 -3
@@ -827,15 +827,16 @@ static int batch_objects(struct batch_options *opt) cb.seen = &seen;for_each_loose_object (batch_unordered_loose, &cb,0 ); -for_each_packed_object (batch_unordered_packed, &cb, -FOR_EACH_OBJECT_PACK_ORDER ); +for_each_packed_object (the_repository, batch_unordered_packed, + &cb,FOR_EACH_OBJECT_PACK_ORDER );oidset_clear (&seen); }else {struct oid_array sa =OID_ARRAY_INIT ;for_each_loose_object (collect_loose_object, &sa,0 ); -for_each_packed_object (collect_packed_object, &sa,0 ); +for_each_packed_object (the_repository, collect_packed_object, + &sa,0 );oid_array_for_each_unique (&sa, batch_object_cb, &cb);
builtin/fsck.c +11 -7
@@ -150,7 +150,7 @@ static int mark_object(struct object *obj, enum object_type type,return 0 ; obj->flags |=REACHABLE ; -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))/* * Further recursion does not need to be performed on this * object since it is a promisor object (so it does not need to
@@ -270,7 +270,7 @@ static void check_reachable_object(struct object *obj)* do a full fsck */ if (!(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */
@@ -391,7 +391,10 @@ static void check_connectivity(void)* traversal. */ for_each_loose_object (mark_loose_unreachable_referents,NULL ,0 ); -for_each_packed_object (mark_packed_unreachable_referents,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_unreachable_referents, +NULL , +0 ); }/* Look up all the requirements, warn about missing objects.. */
@@ -488,7 +491,7 @@ static void fsck_handle_reflog_oid(const char *refname, struct object_id *oid, refname, timestamp); obj->flags |=USED ;mark_object_reachable (obj); - }else if (!is_promisor_object (oid)) { + }else if (!is_promisor_object (the_repository, oid)) {error (_ ("%s: invalid reflog entry %s" ), refname,oid_to_hex (oid)); errors_found |=ERROR_REACHABLE ;
@@ -531,7 +534,7 @@ static int fsck_handle_ref(const char *refname, const char *referent UNUSED, con obj =parse_object (the_repository, oid);if (!obj) { -if (is_promisor_object (oid)) { +if (is_promisor_object (the_repository, oid)) {/* * Increment default_refs anyway, because this is a * valid ref.
@@ -966,7 +969,8 @@ int cmd_fsck(int argc,if (connectivity_only) {for_each_loose_object (mark_loose_for_connectivity,NULL ,0 ); -for_each_packed_object (mark_packed_for_connectivity,NULL ,0 ); +for_each_packed_object (the_repository, + mark_packed_for_connectivity,NULL ,0 ); }else {prepare_alt_odb (the_repository);for (odb = the_repository->objects->odb; odb; odb = odb->next)
@@ -1011,7 +1015,7 @@ int cmd_fsck(int argc, &oid);if (!obj || !(obj->flags &HAS_OBJ )) { -if (is_promisor_object (&oid)) +if (is_promisor_object (the_repository, &oid))continue ;error (_ ("%s: object missing" ),oid_to_hex (&oid)); errors_found |=ERROR_OBJECT ;
builtin/pack-objects.c +5 -2
@@ -3858,7 +3858,8 @@ static void show_object__ma_allow_promisor(struct object *obj, const char *name,* Quietly ignore EXPECTED missing objects. This avoids problems with * staging them now and getting an odd error later. */ -if (!has_object (the_repository, &obj->oid,0 ) &&is_promisor_object (&obj->oid)) +if (!has_object (the_repository, &obj->oid,0 ) && +is_promisor_object (to_pack.repo, &obj->oid))return ;show_object (obj, name, data);
@@ -3927,7 +3928,9 @@ static int add_object_in_unpacked_pack(const struct object_id *oid,static void add_objects_in_unpacked_packs (void ) { - if (for_each_packed_object(add_object_in_unpacked_pack, NULL, + if (for_each_packed_object(to_pack.repo, + add_object_in_unpacked_pack, + NULL, FOR_EACH_OBJECT_PACK_ORDER | FOR_EACH_OBJECT_LOCAL_ONLY | FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS |
builtin/repack.c +1 -1
@@ -404,7 +404,7 @@ static void repack_promisor_objects(const struct pack_objects_args *args,* {type -> existing pack order} ordering when computing deltas instead * of a {type -> size} ordering, which may produce better deltas. */ -for_each_packed_object (write_oid, &cmd, +for_each_packed_object (the_repository, write_oid, &cmd,FOR_EACH_OBJECT_PROMISOR_ONLY );if (cmd.in ==-1 ) {
builtin/rev-list.c +1 -1
@@ -121,7 +121,7 @@ static inline void finish_object__ma(struct object *obj)return ;case MA_ALLOW_PROMISOR : -if (is_promisor_object (&obj->oid)) +if (is_promisor_object (the_repository, &obj->oid))return ;die ("unexpected missing %s object '%s'" ,type_name (obj->type),oid_to_hex (&obj->oid));
commit-graph.c +1 -1
@@ -1960,7 +1960,7 @@ static void fill_oids_from_all_packs(struct write_commit_graph_context *ctx) ctx->progress =start_delayed_progress (_ ("Finding commits for commit graph among packed objects" ), ctx->approx_nr_objects); -for_each_packed_object (add_packed_commits, ctx, +for_each_packed_object (ctx->r, add_packed_commits, ctx,FOR_EACH_OBJECT_PACK_ORDER );if (ctx->progress_done < ctx->approx_nr_objects)display_progress (ctx->progress, ctx->approx_nr_objects);
fsck.c +1 -1
@@ -1295,7 +1295,7 @@ static int fsck_blobs(struct oidset *blobs_found, struct oidset *blobs_done, buf =repo_read_object_file (the_repository, oid, &type, &size);if (!buf) { -if (is_promisor_object (oid)) +if (is_promisor_object (the_repository, oid))continue ; ret |= report(options, oid, OBJ_BLOB, msg_missing,
list-objects.c +2 -2
@@ -75,7 +75,7 @@ static void process_blob(struct traversal_context *ctx,*/ if (ctx->revs->exclude_promisor_objects && !repo_has_object_file (the_repository, &obj->oid) && -is_promisor_object (&obj->oid)) +is_promisor_object (ctx->revs->repo, &obj->oid))return ; pathlen = path->len;
@@ -180,7 +180,7 @@ static void process_tree(struct traversal_context *ctx,* an incomplete list of missing objects. */ if (revs->exclude_promisor_objects && -is_promisor_object (&obj->oid)) +is_promisor_object (revs->repo, &obj->oid))return ;if (!revs->do_not_die_on_missing_objects)
object-store-ll.h +2 -2
@@ -550,7 +550,7 @@ typedef int each_packed_object_fn(const struct object_id *oid,int for_each_object_in_pack (struct packed_git *p,each_packed_object_fn ,void *data,enum for_each_object_flags flags); -int for_each_packed_object (each_packed_object_fn ,void *, -enum for_each_object_flags flags); +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags);#endif /* OBJECT_STORE_LL_H */
packfile.c +7 -7
@@ -2200,15 +2200,15 @@ int for_each_object_in_pack(struct packed_git *p,return r; } -int for_each_packed_object (each_packed_object_fn cb,void *data, -enum for_each_object_flags flags) +int for_each_packed_object (struct repository *repo,each_packed_object_fn cb, +void *data,enum for_each_object_flags flags) {struct packed_git *p;int r =0 ;int pack_errors =0 ; -prepare_packed_git (the_repository); -for (p =get_all_packs (the_repository); p; p = p->next) { +prepare_packed_git (repo); +for (p =get_all_packs (repo); p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ; if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
@@ -2286,14 +2286,14 @@ static int add_promisor_object(const struct object_id *oid,return 0 ; } -int is_promisor_object (const struct object_id *oid) +int is_promisor_object (struct repository *r,const struct object_id *oid) {static struct oidset promisor_objects;static int promisor_objects_prepared;if (!promisor_objects_prepared) { -if (repo_has_promisor_remote (the_repository)) { -for_each_packed_object (add_promisor_object, +if (repo_has_promisor_remote (r)) { +for_each_packed_object (r, add_promisor_object, &promisor_objects,FOR_EACH_OBJECT_PROMISOR_ONLY |FOR_EACH_OBJECT_PACK_ORDER );
packfile.h +1 -1
@@ -201,7 +201,7 @@ int has_object_kept_pack(struct repository *r, const struct object_id *oid,* Return 1 if an object in a promisor packfile is or refers to the given * object, 0 otherwise. */ -int is_promisor_object (const struct object_id *oid); +int is_promisor_object (struct repository *r,const struct object_id *oid);/* * Expose a function for fuzz testing.
promisor-remote.c +1 -1
@@ -283,7 +283,7 @@ void promisor_remote_get_direct(struct repository *repo, }for (i =0 ; i < remaining_nr; i++) { -if (is_promisor_object (&remaining_oids[i])) +if (is_promisor_object (repo, &remaining_oids[i]))die (_ ("could not fetch %s from promisor remote" ),oid_to_hex (&remaining_oids[i])); }
reachable.c +1 -1
@@ -324,7 +324,7 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,if (ignore_in_core_kept_packs) flags |=FOR_EACH_OBJECT_SKIP_IN_CORE_KEPT_PACKS ; - r =for_each_packed_object (add_recent_packed, &data, flags); + r =for_each_packed_object (revs->repo, add_recent_packed, &data, flags); done:oidset_clear (&data.extra_recent_oids);
revision.c +5 -4
@@ -390,7 +390,8 @@ static struct object *get_reference(struct rev_info *revs, const char *name,if (!object) {if (revs->ignore_missing)return NULL ; -if (revs->exclude_promisor_objects &&is_promisor_object (oid)) +if (revs->exclude_promisor_objects && +is_promisor_object (revs->repo, oid))return NULL ;if (revs->do_not_die_on_missing_objects) {oidset_insert (&revs->missing_commits, oid);
@@ -432,7 +433,7 @@ static struct commit *handle_commit(struct rev_info *revs,if (revs->ignore_missing_links || (flags &UNINTERESTING ))return NULL ;if (revs->exclude_promisor_objects && -is_promisor_object (&tag->tagged->oid)) +is_promisor_object (revs->repo, &tag->tagged->oid))return NULL ;if (revs->do_not_die_on_missing_objects && oid) {oidset_insert (&revs->missing_commits, oid);
@@ -1211,7 +1212,7 @@ static int process_parents(struct rev_info *revs, struct commit *commit, revs->do_not_die_on_missing_objects;if (repo_parse_commit_gently (revs->repo, p, gently) <0 ) {if (revs->exclude_promisor_objects && -is_promisor_object (&p->object.oid)) { +is_promisor_object (revs->repo, &p->object.oid)) {if (revs->first_parent_only)break ;continue ;
@@ -3915,7 +3916,7 @@ int prepare_revision_walk(struct rev_info *revs) revs->treesame.name ="treesame" ;if (revs->exclude_promisor_objects) { -for_each_packed_object (mark_uninteresting, revs, +for_each_packed_object (revs->repo, mark_uninteresting, revs,FOR_EACH_OBJECT_PROMISOR_ONLY ); }
tag.c +1 -1
@@ -84,7 +84,7 @@ struct object *deref_tag(struct repository *r, struct object *o, const char *war o =NULL ; }if (!o && warn) { -if (last_oid &&is_promisor_object (last_oid)) +if (last_oid &&is_promisor_object (r, last_oid))return NULL ;if (!warnlen) warnlen =strlen (warn);
Signature · 1 line
2.47.0
PATCH v9 7/10 config: make `delta_base_cache_limit` a non-global variable
In reply to Karthik NayakThe delta_base_cache_limit variable is a global config variable used by multiple subsystems. Let's make this non-global, by adding this variable independently to the subsystems where it is used.
First, add the setting to the repo_settings struct, this provides access to the config in places where the repository is available. Use this in packfile.c.
In index-pack.c we add it to the pack_idx_option struct and its constructor. While the repository struct is available here, it may not be set because git index-pack can be used without a repository.
In gc.c add it to the gc_config struct and also the constructor function. The gc functions currently do not have direct access to a repository struct.
These changes are made to remove the usage of delta_base_cache_limit as a global variable in packfile.c. This brings us one step closer to removing the USE_THE_REPOSITORY_VARIABLE definition in packfile.c which we complete in the next patch.
--- builtin/gc.c | 12 +++++++++++- builtin/index-pack.c | 10 +++++++--- config.c | 5 ----- environment.c | 1 - environment.h | 1 - pack-objects.h | 3 ++- pack-write.c | 1 + pack.h | 2 ++ packfile.c | 10 ++++++++-- repo-settings.c | 5 +++++ repo-settings.h | 3 +++ 11 files changed, 39 insertions(+), 14 deletions(-)
builtin/gc.c +11 -1
@@ -138,6 +138,11 @@ struct gc_config {char *repack_filter_to;unsigned long big_pack_threshold;unsigned long max_delta_cache_size; +/* +* Remove this member from gc_config once repo_settings is passed +* through the callchain. +*/ +size_t delta_base_cache_limit; };#define GC_CONFIG_INIT { \
@@ -153,6 +158,7 @@ struct gc_config { .prune_expire = xstrdup("2.weeks.ago"), \ .prune_worktrees_expire = xstrdup("3.months.ago"), \ .max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \ + .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \ }static void gc_config_release (struct gc_config *cfg)
@@ -168,6 +174,7 @@ static void gc_config(struct gc_config *cfg) {const char *value;char *owned =NULL ; +unsigned long ulongval;if (!git_config_get_value ("gc.packrefs" , &value)) {if (value && !strcmp (value,"notbare" ))
@@ -206,6 +213,9 @@ static void gc_config(struct gc_config *cfg)git_config_get_ulong ("gc.bigpackthreshold" , &cfg->big_pack_threshold);git_config_get_ulong ("pack.deltacachesize" , &cfg->max_delta_cache_size); +if (!git_config_get_ulong ("core.deltabasecachelimit" , &ulongval)) + cfg->delta_base_cache_limit = ulongval; +if (!git_config_get_string ("gc.repackfilter" , &owned)) {free (cfg->repack_filter); cfg->repack_filter = owned;
@@ -416,7 +426,7 @@ static uint64_t estimate_repack_memory(struct gc_config *cfg,* read_sha1_file() (either at delta calculation phase, or * writing phase) also fills up the delta base cache */ - heap += delta_base_cache_limit; + heap += cfg->delta_base_cache_limit;/* and of course pack-objects has its own delta cache */ heap += cfg->max_delta_cache_size;
builtin/index-pack.c +7 -3
@@ -1238,7 +1238,7 @@ static void parse_pack_objects(unsigned char *hash)* recursively checking if the resulting object is used as a base * for some more deltas. */ -static void resolve_deltas (void ) +static void resolve_deltas (struct pack_idx_option *opts) {int i;
@@ -1254,7 +1254,7 @@ static void resolve_deltas(void) nr_ref_deltas + nr_ofs_deltas); nr_dispatched =0 ; - base_cache_limit = delta_base_cache_limit * nr_threads; + base_cache_limit = opts->delta_base_cache_limit * nr_threads;if (nr_threads >1 ||getenv ("GIT_FORCE_THREADS" )) {init_thread ();work_lock ();
@@ -1604,6 +1604,10 @@ static int git_index_pack_config(const char *k, const char *v,else opts->flags &= ~WRITE_REV ; } +if (!strcmp (k,"core.deltabasecachelimit" )) { + opts->delta_base_cache_limit =git_config_ulong (k, v, ctx->kvi); +return 0 ; + }return git_default_config (k, v, ctx, cb); }
@@ -1930,7 +1934,7 @@ int cmd_index_pack(int argc,parse_pack_objects (pack_hash);if (report_end_of_input)write_in_full (2 ,"\0" ,1 ); -resolve_deltas (); +resolve_deltas (&opts);conclude_pack (fix_thin_pack, curr_pack, pack_hash);free (ofs_deltas);free (ref_deltas);
config.c +0 -5
@@ -1515,11 +1515,6 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.deltabasecachelimit" )) { - delta_base_cache_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -1
@@ -51,7 +51,6 @@ enum fsync_method fsync_method = FSYNC_METHOD_DEFAULT;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ;size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ;size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ; -size_t delta_base_cache_limit =96 *1024 *1024 ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
environment.h +0 -1
@@ -165,7 +165,6 @@ extern int zlib_compression_level;extern int pack_compression_level;extern size_t packed_git_window_size;extern size_t packed_git_limit; -extern size_t delta_base_cache_limit;extern unsigned long big_file_threshold;extern unsigned long pack_size_limit_cfg;extern int max_allowed_tree_depth;
pack-objects.h +2 -1
@@ -7,7 +7,8 @@struct repository ; -#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_CACHE_SIZE (256 * 1024 * 1024) +#define DEFAULT_DELTA_BASE_CACHE_LIMIT (96 * 1024 * 1024)#define OE_DFS_STATE_BITS 2#define OE_DEPTH_BITS 12
pack-write.c +1 -0
@@ -21,6 +21,7 @@ void reset_pack_idx_option(struct pack_idx_option *opts)memset (opts,0 ,sizeof (*opts)); opts->version =2 ; opts->off32_limit =0x7fffffff ; + opts->delta_base_cache_limit =DEFAULT_DELTA_BASE_CACHE_LIMIT ; }static int sha1_compare (const void *_a,const void *_b)
pack.h +2 -0
@@ -58,6 +58,8 @@ struct pack_idx_option {*/ int anomaly_alloc, anomaly_nr;uint32_t *anomaly; + +size_t delta_base_cache_limit; };void reset_pack_idx_option (struct pack_idx_option *);
packfile.c +8 -2
@@ -1496,7 +1496,9 @@ void clear_delta_base_cache(void) }static void add_delta_base_cache (struct packed_git *p,off_t base_offset, -void *base,unsigned long base_size,enum object_type type) +void *base,unsigned long base_size, +unsigned long delta_base_cache_limit, +enum object_type type) {struct delta_base_cache_entry *ent;struct list_head *lru, *tmp;
@@ -1698,6 +1700,8 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,int delta_stack_nr =0 , delta_stack_alloc =UNPACK_ENTRY_STACK_PREALLOC ;int base_from_cache =0 ; +prepare_repo_settings (p->repo); +write_pack_access_log (p, obj_offset);/* PHASE 1: drill down to the innermost base object */
@@ -1878,7 +1882,9 @@ void *unpack_entry(struct repository *r, struct packed_git *p, off_t obj_offset,* before we are done using it. */ if (!external_base) -add_delta_base_cache (p, base_obj_offset, base, base_size, type); +add_delta_base_cache (p, base_obj_offset, base, base_size, + p->repo->settings.delta_base_cache_limit, + type);free (delta_data);free (external_base);
repo-settings.c +5 -0
@@ -3,6 +3,7 @@#include "repo-settings.h" #include "repository.h" #include "midx.h" +#include "pack-objects.h" static void repo_cfg_bool (struct repository *r,const char *key,int *dest,int def)
@@ -26,6 +27,7 @@ void prepare_repo_settings(struct repository *r)const char *strval;int manyfiles;int read_changed_paths; +unsigned long ulongval;if (!r->gitdir)BUG ("Cannot add settings for uninitialized repository" );
@@ -123,6 +125,9 @@ void prepare_repo_settings(struct repository *r)* removed. */ r->settings.command_requires_full_index =1 ; + +if (!repo_config_get_ulong (r,"core.deltabasecachelimit" , &ulongval)) + r->settings.delta_base_cache_limit = ulongval; }enum log_refs_config repo_settings_get_log_all_ref_updates (struct repository *repo)
repo-settings.h +3 -0
@@ -57,12 +57,15 @@ struct repo_settings {int core_multi_pack_index;int warn_ambiguous_refs;/* lazily loaded via accessor */ + +size_t delta_base_cache_limit; };#define REPO_SETTINGS_INIT { \ .index_version = -1, \ .core_untracked_cache = UNTRACKED_CACHE_KEEP, \ .fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \ .warn_ambiguous_refs = -1, \ + .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \ }void prepare_repo_settings (struct repository *r);
Signature · 1 line
2.47.0
PATCH v9 8/10 config: make `packed_git_(limit|window_size)` non-global variables
In reply to Karthik NayakThe variables packed_git_window_size and packed_git_limit are global config variables used in the packfile.c file. Since it is only used in this file, let's change it from being a global config variable to a local variable for the subsystem.
With this, we rid packfile.c from all global variable usage and this means we can also remove the USE_THE_REPOSITORY_VARIABLE guard from the file.
--- builtin/fast-import.c | 4 ++-- config.c | 17 ----------------- environment.c | 2 -- packfile.c | 23 +++++++++++++++-------- packfile.h | 2 +- repo-settings.c | 13 +++++++++++++ repo-settings.h | 4 ++++ 7 files changed, 35 insertions(+), 30 deletions(-)
builtin/fast-import.c +2 -2
@@ -3539,7 +3539,7 @@ static void parse_argv(void)int cmd_fast_import (int argc,const char **argv,const char *prefix, - struct repository *repo UNUSED) +struct repository *repo) {unsigned int i;
@@ -3660,7 +3660,7 @@ int cmd_fast_import(int argc,fprintf (stderr," pools: %10lu KiB\n" , (unsigned long )((tree_entry_allocd + fi_mem_pool.pool_alloc) /1024 ));fprintf (stderr," objects: %10" PRIuMAX" KiB\n" , (alloc_count*sizeof (struct object_entry ))/1024 );fprintf (stderr,"---------------------------------------------------------------------\n" ); -pack_report (); +pack_report (repo);fprintf (stderr,"---------------------------------------------------------------------\n" );fprintf (stderr,"\n" ); }
config.c +0 -17
@@ -1493,28 +1493,11 @@ static int git_default_core_config(const char *var, const char *value,return 0 ; } -if (!strcmp (var,"core.packedgitwindowsize" )) { -int pgsz_x2 =getpagesize () *2 ; - packed_git_window_size =git_config_ulong (var, value, ctx->kvi); - -/* This value must be multiple of (pagesize * 2) */ - packed_git_window_size /= pgsz_x2; -if (packed_git_window_size <1 ) - packed_git_window_size =1 ; - packed_git_window_size *= pgsz_x2; -return 0 ; - } -if (!strcmp (var,"core.bigfilethreshold" )) { big_file_threshold =git_config_ulong (var, value, ctx->kvi);return 0 ; } -if (!strcmp (var,"core.packedgitlimit" )) { - packed_git_limit =git_config_ulong (var, value, ctx->kvi); -return 0 ; - } -if (!strcmp (var,"core.autocrlf" )) {if (value && !strcasecmp (value,"input" )) { auto_crlf =AUTO_CRLF_INPUT ;
environment.c +0 -2
@@ -49,8 +49,6 @@ int fsync_object_files = -1;int use_fsync =-1 ;enum fsync_method fsync_method =FSYNC_METHOD_DEFAULT ;enum fsync_component fsync_components =FSYNC_COMPONENTS_DEFAULT ; -size_t packed_git_window_size =DEFAULT_PACKED_GIT_WINDOW_SIZE ; -size_t packed_git_limit =DEFAULT_PACKED_GIT_LIMIT ;unsigned long big_file_threshold =512 *1024 *1024 ;char *editor_program;char *askpass_program;
packfile.c +15 -8
@@ -1,4 +1,3 @@ -#define USE_THE_REPOSITORY_VARIABLE #include "git-compat-util.h" #include "environment.h"
@@ -46,15 +45,15 @@ static size_t pack_mapped;#define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; } -void pack_report (void ) +void pack_report (struct repository *repo) {fprintf (stderr,"pack_report: getpagesize() = %10" SZ_FMT "\n" "pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" "pack_report: core.packedGitLimit = %10" SZ_FMT "\n" ,sz_fmt (getpagesize ()), -sz_fmt (packed_git_window_size), -sz_fmt (packed_git_limit)); +sz_fmt (repo->settings.packed_git_window_size), +sz_fmt (repo->settings.packed_git_limit));fprintf (stderr,"pack_report: pack_used_ctr = %10u\n" "pack_report: pack_mmap_calls = %10u\n"
@@ -650,8 +649,15 @@ unsigned char *use_pack(struct packed_git *p,break ; }if (!win) { -size_t window_align = packed_git_window_size /2 ; +size_t window_align;off_t len; +struct repo_settings *settings; + +/* lazy load the settings in case it hasn't been setup */ +prepare_repo_settings (p->repo); + settings = &p->repo->settings; + + window_align = settings->packed_git_window_size /2 ;if (p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name);
@@ -659,11 +665,12 @@ unsigned char *use_pack(struct packed_git *p,CALLOC_ARRAY (win,1 ); win->offset = (offset / window_align) * window_align; len = p->pack_size - win->offset; -if (len > packed_git_window_size) - len = packed_git_window_size; +if (len > settings->packed_git_window_size) + len = settings->packed_git_window_size; win->len = (size_t )len; pack_mapped += win->len; -while (packed_git_limit < pack_mapped + +while (settings->packed_git_limit < pack_mapped &&unuse_one_window (p)) ;/* nothing */ win->base = xmmap_gently(NULL, win->len,
packfile.h +1 -1
@@ -89,7 +89,7 @@ unsigned long repo_approximate_object_count(struct repository *r);struct packed_git *find_oid_pack (const struct object_id *oid,struct packed_git *packs); -void pack_report (void ); +void pack_report (struct repository *repo);/* * mmap the index file for the specified packfile (if it is not
repo-settings.c +13 -0
@@ -128,6 +128,19 @@ void prepare_repo_settings(struct repository *r)if (!repo_config_get_ulong (r,"core.deltabasecachelimit" , &ulongval)) r->settings.delta_base_cache_limit = ulongval; + +if (!repo_config_get_ulong (r,"core.packedgitwindowsize" , &ulongval)) { +int pgsz_x2 =getpagesize () *2 ; + +/* This value must be multiple of (pagesize * 2) */ + ulongval /= pgsz_x2; +if (ulongval <1 ) + ulongval =1 ; + r->settings.packed_git_window_size = ulongval * pgsz_x2; + } + +if (!repo_config_get_ulong (r,"core.packedgitlimit" , &ulongval)) + r->settings.packed_git_limit = ulongval; }enum log_refs_config repo_settings_get_log_all_ref_updates (struct repository *repo)
repo-settings.h +4 -0
@@ -59,6 +59,8 @@ struct repo_settings {int warn_ambiguous_refs;/* lazily loaded via accessor */ size_t delta_base_cache_limit; +size_t packed_git_window_size; +size_t packed_git_limit; };#define REPO_SETTINGS_INIT { \ .index_version = -1, \
@@ -66,6 +68,8 @@ struct repo_settings { .fetch_negotiation_algorithm = FETCH_NEGOTIATION_CONSECUTIVE, \ .warn_ambiguous_refs = -1, \ .delta_base_cache_limit = DEFAULT_DELTA_BASE_CACHE_LIMIT, \ + .packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE, \ + .packed_git_limit = DEFAULT_PACKED_GIT_LIMIT, \ }void prepare_repo_settings (struct repository *r);
Signature · 1 line
2.47.0
PATCH v9 9/10 midx: add repository to `multi_pack_index` struct
In reply to Karthik NayakThe multi_pack_index struct represents the MIDX for a repository. Here, we add a pointer to the repository in this struct, allowing direct use of the repository variable without relying on the global the_repository struct.
With this addition, we can determine the repository associated with a bitmap_index struct. A bitmap_index points to either a packed_git or a multi_pack_index, both of which have direct repository references. To support this, we introduce a static helper function, bitmap_repo, in pack-bitmap.c, which retrieves a repository given a bitmap_index.
With this, we clear up all usages of the_repository within pack-bitmap.c and also remove the USE_THE_REPOSITORY_VARIABLE definition. Bringing us another step closer to remove all global variable usage.
Although this change also opens up the potential to clean up midx.c, doing so would require additional refactoring to pass the repository struct to functions where the MIDX struct is created: a task better suited for future patches.
--- midx.c | 1 + midx.h | 3 ++ pack-bitmap.c | 90 +++++++++++++++++++++++++++++++-------------------- 3 files changed, 59 insertions(+), 35 deletions(-)
midx.c +1 -0
@@ -131,6 +131,7 @@ static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir m->data = midx_map; m->data_len = midx_size; m->local = local; + m->repo = the_repository; m->signature =get_be32 (m->data);if (m->signature !=MIDX_SIGNATURE )
midx.h +3 -0
@@ -71,6 +71,9 @@ struct multi_pack_index {const char **pack_names;struct packed_git **packs; + +struct repository *repo; +char object_dir[FLEX_ARRAY ]; };
pack-bitmap.c +55 -35
251 lines not on this page · raw message
Signature · 1 line
2.47.0
PATCH v9 10/10 packfile.c: remove unnecessary prepare_packed_git() call
In reply to Karthik NayakFrom: Taylor Blau <[email protected]>
In 454ea2e4d7 (treewide: use get_all_packs, 2018-08-20) we converted existing calls to both:
- get_packed_git(), as well as
- the_repository->objects->packed_git
, to instead use the new get_all_packs() function.
In the instance that this commit addresses, there was a preceding call to prepare_packed_git(), which dates all the way back to 660c889e46 (sha1_file: add for_each iterators for loose and packed objects, 2014-10-15) when its caller (for_each_packed_object()) was first introduced.
This call could have been removed in 454ea2e4d7, since get_all_packs() itself calls prepare_packed_git(). But the translation in 454ea2e4d7 was (to the best of my knowledge) a find-and-replace rather than inspecting each individual caller.
Having an extra prepare_packed_git() call here is harmless, since it will notice that we have already set the 'packed_git_initialized' field and the call will be a noop. So we're only talking about a few dozen CPU cycles to set up and tear down the stack frame.
But having a lone prepare_packed_git() call immediately before a call to get_all_packs() confused me, so let's remove it as redundant to avoid more confusion in the future.
--- packfile.c | 1 - 1 file changed, 1 deletion(-)
packfile.c +0 -1
@@ -2220,7 +2220,6 @@ int for_each_packed_object(struct repository *repo, each_packed_object_fn cb,int r =0 ;int pack_errors =0 ; -prepare_packed_git (repo);for (p =get_all_packs (repo); p; p = p->next) {if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local)continue ;
Signature · 1 line
2.47.0
v10 cover packfile: avoid using the 'the_repository' global variable
In reply to Karthik NayakThe packfile.c file uses the global variable 'the_repository' extensively throughout the code. Let's remove all usecases of this, by modifying the required functions to accept a 'struct repository' instead. This is to clean up usage of global state.
The first 3 patches are mostly internal to packfile.c, we add the repository field to the packed_git struct and this is used to clear up some useages of the global variables.
The next 3 patches are more disruptive, they modify the function definition of odb_pack_name, has_object[_kept]_pack and for_each_packed_object to receive a repository, helping remove other usages of 'the_repository' variable.
Finally, the next two patches deal with global config values. These values are localized. The last patch is removal of an unecessary call to prepare_packed_git(). For v5 onwards, I've rebased the series off the master: 8f8d6eee53 (The seventh batch, 2024-11-01), as a dependency for this series 'jk/dumb-http-finalize' was merged to master. I've found no conflicts while merging with seen & next. But since this series does touch multiple files, there could be future conflicts.
Changes in v10:
- Grammar corrections in the commit messages.
Changes in v9:
- Added a comment in gc_config to indicate that eventually the delta_base_cache_limit variable should be used through repo_settings.
Changes in v8:
- Fix typos in comments
- For packfile.c use delta_base_cache_limit from the repository settings, this avoids loading the config in hot paths.
- Rename longval to ulongval to better signify the type.
Changes in v7:
- Cleanup stale commit message.
- Add missing space in if statement.
- Fix typo s/incase/in case/.
Changes in v6:
- Lazy load repository settings in packfile.c. This ensures that the settings are available for sure and we do not rely on callees setting it.
- Use size_t for delta_base_cache_limit.
Changes in v5:
- Move packed_git* settings to repo_settings to ensure we don't keep reparsing the settings in use_pack.
Changes in v4:
- Renamed the repository field within packed_git and multi_pack_index from r to repo, while keeping function parameters to be r.
- Fixed bad braces.
Changes in v3:
- Improved commit messages. In the first commit to talk about how packed_git struct could also be part of the alternates of a repository. In the 7th commit to talk about the motive behind removing the global variable.
- Changed 'packed_git->repo' to 'packed_git->r' to keep it consistent with the rest of the code base.
- Replaced 'the_repository' with locally available access to the repository struct in multiple regions.
- Removed unecessary inclusion of the 'repository.h' header file by forward declaring the 'repository' struct.
- Replace memcpy with hashcpy.
- Change the logic in the 7th patch to use if else statements.
- Added an extra commit to cleanup pack-bitmap.c.
Karthik Nayak (9):
packfile: add repository to struct packed_git
packfile: use repository from packed_git directly
packfile: pass repository to static function in the file
packfile: pass down repository to odb_pack_name
packfile: pass down repository to has_object[_kept]_pack
packfile: pass down repository to for_each_packed_object
config: make delta_base_cache_limit a non-global variable
config: make packed_git_(limit|window_size) non-global variables
midx: add repository to multi_pack_index struct
Taylor Blau (1):
packfile.c: remove unnecessary prepare_packed_git() call
builtin/cat-file.c | 7 +- builtin/count-objects.c | 2 +- builtin/fast-import.c | 15 ++-- builtin/fsck.c | 20 +++--- builtin/gc.c | 12 +++- builtin/index-pack.c | 20 ++++-- builtin/pack-objects.c | 11 +-- builtin/pack-redundant.c | 2 +- builtin/repack.c | 2 +- builtin/rev-list.c | 2 +- commit-graph.c | 4 +- config.c | 22 ------ connected.c | 3 +- diff.c | 3 +- environment.c | 3 - environment.h | 1 - fsck.c | 2 +- http.c | 4 +- list-objects.c | 7 +- midx-write.c | 2 +- midx.c | 3 +- midx.h | 3 + object-store-ll.h | 9 ++- pack-bitmap.c | 90 ++++++++++++++---------- pack-objects.h | 3 +- pack-write.c | 1 + pack.h | 2 + packfile.c | 144 ++++++++++++++++++++++----------------- packfile.h | 18 +++-- promisor-remote.c | 2 +- prune-packed.c | 2 +- reachable.c | 4 +- repo-settings.c | 18 +++++ repo-settings.h | 7 ++ revision.c | 13 ++-- tag.c | 2 +- 36 files changed, 275 insertions(+), 190 deletions(-)
Range-diff against v9:
1: d1fdd6996a ! 1: d6d571c58e packfile: add repository to struct packed_git
@@ Commit message
on the global the_repository object in packfile.c by simply using repository information now readily available in the struct.
- We do need to consider that a pack file could be part of the alternates
+ We do need to consider that a packfile could be part of the alternates
of a repository, but considering that we only have one repository struct
- and also that we currently anyways use 'the_repository'. We should be
+ and also that we currently anyways use 'the_repository', we should be
OK with this change.
We also modify alloc_packed_git to ensure that the repository is added
2: 65c09858ce = 2: fa69763468 packfile: use `repository` from `packed_git` directly 3: 80632934d1 ! 3: c6acbece46 packfile: pass `repository` to static function in the file
@@ Commit message
packfile: pass repository to static function in the file
Some of the static functions in the packfile.c access global
- variables, which can simply be avoiding by passing the repository
+ variables, which can simply be avoided by passing the repository
struct down to them. Let's do that.
4: 67d71eab83 = 4: a8588d6086 packfile: pass down repository to `odb_pack_name` 5: ee210fa153 = 5: b3fe20c8f1 packfile: pass down repository to `has_object[_kept]_pack` 6: 8db7094f4e = 6: ad46b339ea packfile: pass down repository to `for_each_packed_object` 7: a66494384d = 7: 342a26572d config: make `delta_base_cache_limit` a non-global variable 8: bce9196f6b = 8: 6e55daf5b3 config: make `packed_git_(limit|window_size)` non-global variables 9: c7fba8cf6a = 9: 6e0ec955e6 midx: add repository to `multi_pack_index` struct 10: d7f475fbd0 = 10: e33fa2ea0d packfile.c: remove unnecessary prepare_packed_git() call
Signature · 1 line
2.47.1
PATCH v10 1/10 packfile: add repository to struct `packed_git`
The struct packed_git holds information regarding a packed object file. Let's add the repository variable to this object, to represent the repository that this packfile belongs to. This helps remove dependency on the global the_repository object in packfile.c by simply using repository information now readily available in the struct.
We do need to consider that a packfile could be part of the alternates of a repository, but considering that we only have one repository struct and also that we currently anyways use 'the_repository', we should be OK with this change.
We also modify alloc_packed_git to ensure that the repository is added to newly created packed_git structs. This requires modifying the function and all its callee to pass the repository object down the levels.
--- builtin/fast-import.c | 3 ++- builtin/index-pack.c | 6 ++++-- commit-graph.c | 2 +- connected.c | 3 ++- http.c | 2 +- midx-write.c | 2 +- midx.c | 2 +- object-store-ll.h | 5 +++++ packfile.c | 15 +++++++++------ packfile.h | 6 ++++-- 10 files changed, 30 insertions(+), 16 deletions(-)
builtin/fast-import.c +2 -1
@@ -765,6 +765,7 @@ static void start_packfile(void) p->pack_fd = pack_fd; p->do_not_close =1 ; + p->repo = the_repository; pack_file =hashfd (pack_fd, p->pack_name); pack_data = p;
@@ -888,7 +889,7 @@ static void end_packfile(void) idx_name =keep_pack (create_index ());/* Register the packfile with core git's machinery. */ - new_p =add_packed_git (idx_name,strlen (idx_name),1 ); + new_p =add_packed_git (pack_data->repo, idx_name,strlen (idx_name),1 );if (!new_p)die ("core git rejected index %s" , idx_name); all_packs[pack_id] = new_p;
builtin/index-pack.c +4 -2
@@ -1552,7 +1552,8 @@ static void final(const char *final_pack_name, const char *curr_pack_name,if (do_fsck_object) {struct packed_git *p; - p =add_packed_git (final_index_name,strlen (final_index_name),0 ); + p =add_packed_git (the_repository, final_index_name, +strlen (final_index_name),0 );if (p)install_packed_git (the_repository, p); }
@@ -1650,7 +1651,8 @@ static void read_v2_anomalous_offsets(struct packed_git *p,static void read_idx_option (struct pack_idx_option *opts,const char *pack_name) { -struct packed_git *p =add_packed_git (pack_name,strlen (pack_name),1 ); +struct packed_git *p =add_packed_git (the_repository, pack_name, +strlen (pack_name),1 );if (!p)die (_ ("Cannot open existing pack file '%s'" ), pack_name);
commit-graph.c +1 -1
@@ -1914,7 +1914,7 @@ static int fill_oids_from_packs(struct write_commit_graph_context *ctx,struct packed_git *p;strbuf_setlen (&packname, dirlen);strbuf_addstr (&packname, pack_indexes->items[i].string); - p =add_packed_git (packname.buf, packname.len,1 ); + p =add_packed_git (ctx->r, packname.buf, packname.len,1 );if (!p) { ret =error (_ ("error adding pack %s" ), packname.buf); goto cleanup;
connected.c +2 -1
@@ -54,7 +54,8 @@ int check_connected(oid_iterate_fn fn, void *cb_data,strbuf_add (&idx_file, transport->pack_lockfiles.items[0 ].string, base_len);strbuf_addstr (&idx_file,".idx" ); - new_pack =add_packed_git (idx_file.buf, idx_file.len,1 ); + new_pack =add_packed_git (the_repository, idx_file.buf, + idx_file.len,1 );strbuf_release (&idx_file); }
http.c +1 -1
@@ -2439,7 +2439,7 @@ static int fetch_and_setup_pack_index(struct packed_git **packs_head,if (!tmp_idx)return -1 ; - new_pack =parse_pack_index (sha1, tmp_idx); + new_pack =parse_pack_index (the_repository, sha1, tmp_idx);if (!new_pack) {unlink (tmp_idx);free (tmp_idx);
midx-write.c +1 -1
@@ -154,7 +154,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len,return ;ALLOC_GROW (ctx->info, ctx->nr +1 , ctx->alloc); - p =add_packed_git (full_path, full_path_len,0 ); + p =add_packed_git (the_repository, full_path, full_path_len,0 );if (!p) {warning (_ ("failed to add packfile '%s'" ), full_path);
midx.c +1 -1
@@ -464,7 +464,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m,strhash (key.buf), key.buf, struct packed_git, packmap_ent);if (!p) { - p =add_packed_git (pack_name.buf, pack_name.len, m->local); + p =add_packed_git (r, pack_name.buf, pack_name.len, m->local);if (p) {install_packed_git (r, p);list_add_tail (&p->mru, &r->objects->packed_git_mru);
object-store-ll.h +5 -0
@@ -10,6 +10,7 @@struct oidmap ;struct oidtree ;struct strbuf ; +struct repository ;struct object_directory {struct object_directory *next;
@@ -135,6 +136,10 @@ struct packed_git {*/ const uint32_t *mtimes_map;size_t mtimes_size; + +/* repo denotes the repository this packfile belongs to */ +struct repository *repo; +/* something like ".git/objects/pack/xxxxx.pack" */ char pack_name[FLEX_ARRAY ];/* more */ };
packfile.c +9 -6
@@ -217,11 +217,12 @@ uint32_t get_pack_fanout(struct packed_git *p, uint32_t value)return ntohl (level1_ofs[value]); } -static struct packed_git *alloc_packed_git (int extra) +static struct packed_git *alloc_packed_git (struct repository *r,int extra) {struct packed_git *p =xmalloc (st_add (sizeof (*p), extra));memset (p,0 ,sizeof (*p)); p->pack_fd =-1 ; + p->repo = r;return p; }
@@ -233,11 +234,12 @@ static char *pack_path_from_idx(const char *idx_path)return xstrfmt ("%.*s.pack" , (int )len, idx_path); } -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path) {char *path =pack_path_from_idx (idx_path);size_t alloc =st_add (strlen (path),1 ); -struct packed_git *p =alloc_packed_git (alloc); +struct packed_git *p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path);
@@ -703,7 +705,8 @@ void unuse_pack(struct pack_window **w_cursor) } } -struct packed_git *add_packed_git (const char *path,size_t path_len,int local) +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local) {struct stat st;size_t alloc;
@@ -721,7 +724,7 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)* the use xsnprintf double-checks that) */ alloc =st_add3 (path_len,strlen (".promisor" ),1 ); - p =alloc_packed_git (alloc); + p =alloc_packed_git (r, alloc);memcpy (p->pack_name, path, path_len);xsnprintf (p->pack_name + path_len, alloc - path_len,".keep" );
@@ -877,7 +880,7 @@ static void prepare_pack(const char *full_name, size_t full_name_len,/* Don't reopen a pack we already have. */ if (!hashmap_get (&data->r->objects->pack_map, &hent, pack_name)) { - p =add_packed_git (full_name, full_name_len, data->local); + p =add_packed_git (data->r, full_name, full_name_len, data->local);if (p)install_packed_git (data->r, p); }
packfile.h +4 -2
@@ -46,7 +46,8 @@ const char *pack_basename(struct packed_git *p);* and does not add the resulting packed_git struct to the internal list of * packs. You probably want add_packed_git() instead. */ -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +struct packed_git *parse_pack_index (struct repository *r,unsigned char *sha1, +const char *idx_path);typedef void each_file_in_pack_dir_fn (const char *full_path,size_t full_path_len,const char *file_name,void *data);
@@ -113,7 +114,8 @@ void close_pack(struct packed_git *);void close_object_store (struct raw_object_store *o);void unuse_pack (struct pack_window **);void clear_delta_base_cache (void ); -struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +struct packed_git *add_packed_git (struct repository *r,const char *path, +size_t path_len,int local);/* * Unlink the .pack and associated extension files.
Signature · 1 line
2.47.1
PATCH v10 2/10 packfile: use `repository` from `packed_git` directly
In reply to Karthik NayakIn the previous commit, we introduced the repository structure inside packed_git. This provides an alternative route instead of using the global the_repository variable. Let's modify packfile.c now to use this field wherever possible instead of relying on the global state. There are still a few instances of the_repository usage in the file, where there is no struct packed_git locally available, which will be fixed in the following commits.
--- packfile.c | 50 +++++++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 23 deletions(-)
packfile.c +27 -23
@@ -79,7 +79,7 @@ static int check_packed_git_idx(const char *path, struct packed_git *p)size_t idx_size;int fd =git_open (path), ret;struct stat st; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (fd <0 )return -1 ;
@@ -243,7 +243,7 @@ struct packed_git *parse_pack_index(struct repository *r, unsigned char *sha1,memcpy (p->pack_name, path, alloc);/* includes NUL */ free (path); -hashcpy (p->hash, sha1, the_repository->hash_algo); +hashcpy (p->hash, sha1, p->repo->hash_algo);if (check_packed_git_idx (idx_path, p)) {free (p);return NULL ;
@@ -278,7 +278,7 @@ static int unuse_one_window(struct packed_git *current)if (current)scan_windows (current, &lru_p, &lru_w, &lru_l); -for (p = the_repository->objects->packed_git; p; p = p->next) +for (p = current->repo->objects->packed_git; p; p = p->next)scan_windows (p, &lru_p, &lru_w, &lru_l);if (lru_p) {munmap (lru_w->base, lru_w->len);
@@ -540,7 +540,7 @@ static int open_packed_git_1(struct packed_git *p)unsigned char hash[GIT_MAX_RAWSZ ];unsigned char *idx_hash;ssize_t read_result; -const unsigned hashsz = the_hash_algo->rawsz; +const unsigned hashsz = p->repo->hash_algo->rawsz;if (open_pack_index (p))return error ("packfile %s index unavailable" , p->pack_name);
@@ -597,7 +597,7 @@ static int open_packed_git_1(struct packed_git *p)if (read_result != hashsz)return error ("packfile %s signature is unavailable" , p->pack_name); idx_hash = ((unsigned char *)p->index_data) + p->index_size - hashsz *2 ; -if (!hasheq (hash, idx_hash, the_repository->hash_algo)) +if (!hasheq (hash, idx_hash, p->repo->hash_algo))return error ("packfile %s does not match index" , p->pack_name);return 0 ; }
@@ -637,7 +637,7 @@ unsigned char *use_pack(struct packed_git *p,*/ if (!p->pack_size && p->pack_fd ==-1 &&open_packed_git (p))die ("packfile %s cannot be accessed" , p->pack_name); -if (offset > (p->pack_size - the_hash_algo->rawsz)) +if (offset > (p->pack_size - p->repo->hash_algo->rawsz))die ("offset beyond end of packfile (truncated pack?)" );if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" ));
@@ -711,6 +711,7 @@ struct packed_git *add_packed_git(struct repository *r, const char *path,struct stat st;size_t alloc;struct packed_git *p; +struct object_id oid;/* * Make sure a corresponding .pack file exists and that
@@ -751,9 +752,13 @@ struct packed_git *add_packed_git(struct repository *r, const char *path, p->pack_size = st.st_size; p->pack_local = local; p->mtime = st.st_mtime; -if (path_len < the_hash_algo->hexsz || -get_hash_hex (path + path_len - the_hash_algo->hexsz, p->hash)) -hashclr (p->hash, the_repository->hash_algo); +if (path_len < r->hash_algo->hexsz || +get_oid_hex_algop (path + path_len - r->hash_algo->hexsz, &oid, + r->hash_algo)) +hashclr (p->hash, r->hash_algo); +else +hashcpy (p->hash, oid.hash, r->hash_algo); +return p; }
@@ -1243,9 +1248,9 @@ off_t get_delta_base(struct packed_git *p, }else if (type ==OBJ_REF_DELTA ) {/* The base entry _must_ be in the same pack */ struct object_id oid; -oidread (&oid, base_info, the_repository->hash_algo); +oidread (&oid, base_info, p->repo->hash_algo); base_offset =find_pack_entry_one (&oid, p); - *curpos += the_hash_algo->rawsz; + *curpos += p->repo->hash_algo->rawsz; }else die ("I am totally screwed" );return base_offset;
@@ -1266,7 +1271,7 @@ static int get_delta_base_oid(struct packed_git *p, {if (type ==OBJ_REF_DELTA ) {unsigned char *base =use_pack (p, w_curs, curpos,NULL ); -oidread (oid, base, the_repository->hash_algo); +oidread (oid, base, p->repo->hash_algo);return 0 ; }else if (type ==OBJ_OFS_DELTA ) {uint32_t base_pos;
@@ -1608,7 +1613,7 @@ int packed_object_info(struct repository *r, struct packed_git *p, goto out; } }else -oidclr (oi->delta_base_oid, the_repository->hash_algo); +oidclr (oi->delta_base_oid, p->repo->hash_algo); } oi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :
@@ -1897,7 +1902,7 @@ int bsearch_pack(const struct object_id *oid, const struct packed_git *p, uint32 {const unsigned char *index_fanout = p->index_data;const unsigned char *index_lookup; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;int index_lookup_width;if (!index_fanout)
@@ -1922,7 +1927,7 @@ int nth_packed_object_id(struct object_id *oid, uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz;if (!index) {if (open_pack_index (p))return -1 ;
@@ -1933,11 +1938,10 @@ int nth_packed_object_id(struct object_id *oid, index +=4 *256 ;if (p->index_version ==1 ) {oidread (oid, index +st_add (st_mult (hashsz +4 , n),4 ), - the_repository->hash_algo); + p->repo->hash_algo); }else { index +=8 ; -oidread (oid, index +st_mult (hashsz, n), - the_repository->hash_algo); +oidread (oid, index +st_mult (hashsz, n), p->repo->hash_algo); }return 0 ; }
@@ -1959,7 +1963,7 @@ void check_pack_index_ptr(const struct packed_git *p, const void *vptr)off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) {const unsigned char *index = p->index_data; -const unsigned int hashsz = the_hash_algo->rawsz; +const unsigned int hashsz = p->repo->hash_algo->rawsz; index +=4 *256 ;if (p->index_version ==1 ) {return ntohl (*((uint32_t *)(index +st_mult (hashsz +4 , n))));
@@ -2159,7 +2163,7 @@ int for_each_object_in_pack(struct packed_git *p,int r =0 ;if (flags &FOR_EACH_OBJECT_PACK_ORDER ) { -if (load_pack_revindex (the_repository, p)) +if (load_pack_revindex (p->repo, p))return -1 ; }
@@ -2227,7 +2231,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, }static int add_promisor_object (const struct object_id *oid, - struct packed_git *pack UNUSED, +struct packed_git *pack, uint32_t pos UNUSED,void *set_) {
@@ -2235,12 +2239,12 @@ static int add_promisor_object(const struct object_id *oid,struct object *obj;int we_parsed_object; - obj =lookup_object (the_repository, oid); + obj =lookup_object (pack->repo, oid);if (obj && obj->parsed) { we_parsed_object =0 ; }else { we_parsed_object =1 ; - obj =parse_object (the_repository, oid); + obj =parse_object (pack->repo, oid); }if (!obj)
Signature · 1 line
2.47.1
PATCH v10 3/10 packfile: pass `repository` to static function in the file
In reply to Karthik NayakSome of the static functions in the packfile.c access global variables, which can simply be avoided by passing the repository struct down to them. Let's do that.
--- packfile.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
packfile.c +8 -7
@@ -460,13 +460,13 @@ static void find_lru_pack(struct packed_git *p, struct packed_git **lru_p, struc *accept_windows_inuse = has_windows_inuse; } -static int close_one_pack (void ) +static int close_one_pack (struct repository *r) {struct packed_git *p, *lru_p =NULL ;struct pack_window *mru_w =NULL ;int accept_windows_inuse =1 ; -for (p = the_repository->objects->packed_git; p; p = p->next) { +for (p = r->objects->packed_git; p; p = p->next) {if (p->pack_fd ==-1 )continue ;find_lru_pack (p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -555,7 +555,7 @@ static int open_packed_git_1(struct packed_git *p) pack_max_fds =1 ; } -while (pack_max_fds <= pack_open_fds &&close_one_pack ()) +while (pack_max_fds <= pack_open_fds &&close_one_pack (p->repo)) ;/* nothing */ p->pack_fd =git_open (p->pack_name);
@@ -610,7 +610,8 @@ static int open_packed_git(struct packed_git *p)return -1 ; } -static int in_window (struct pack_window *win,off_t offset) +static int in_window (struct repository *r,struct pack_window *win, +off_t offset) {/* We must promise at least one full hash after the * offset is available from this window, otherwise the offset
@@ -620,7 +621,7 @@ static int in_window(struct pack_window *win, off_t offset)*/ off_t win_off = win->offset;return win_off <= offset - && (offset + the_hash_algo->rawsz) <= (win_off + win->len); + && (offset + r->hash_algo->rawsz) <= (win_off + win->len); } unsigned char *use_pack(struct packed_git *p,
@@ -642,11 +643,11 @@ unsigned char *use_pack(struct packed_git *p,if (offset <0 )die (_ ("offset before end of packfile (broken .idx?)" )); -if (!win || !in_window (win, offset)) { +if (!win || !in_window (p->repo, win, offset)) {if (win) win->inuse_cnt--;for (win = p->windows; win; win = win->next) { -if (in_window (win, offset)) +if (in_window (p->repo, win, offset))break ; }if (!win) {
Signature · 1 line
2.47.1
PATCH v10 4/10 packfile: pass down repository to `odb_pack_name`
In reply to Karthik NayakThe function odb_pack_name currently relies on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers.
--- builtin/fast-import.c | 8 ++++---- builtin/index-pack.c | 4 ++-- builtin/pack-redundant.c | 2 +- http.c | 2 +- packfile.c | 9 ++++----- packfile.h | 3 ++- 6 files changed, 14 insertions(+), 14 deletions(-)
builtin/fast-import.c +4 -4
@@ -806,7 +806,7 @@ static char *keep_pack(const char *curr_index_name)struct strbuf name =STRBUF_INIT ;int keep_fd; -odb_pack_name (&name, pack_data->hash,"keep" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"keep" ); keep_fd =odb_pack_keep (name.buf);if (keep_fd <0 )die_errno ("cannot create keep file" );
@@ -814,11 +814,11 @@ static char *keep_pack(const char *curr_index_name)if (close (keep_fd))die_errno ("failed to write keep file" ); -odb_pack_name (&name, pack_data->hash,"pack" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"pack" );if (finalize_object_file (pack_data->pack_name, name.buf))die ("cannot store pack file" ); -odb_pack_name (&name, pack_data->hash,"idx" ); +odb_pack_name (pack_data->repo, &name, pack_data->hash,"idx" );if (finalize_object_file (curr_index_name, name.buf))die ("cannot store index file" );free ((void *)curr_index_name);
@@ -832,7 +832,7 @@ static void unkeep_all_packs(void)for (k =0 ; k < pack_id; k++) {struct packed_git *p = all_packs[k]; -odb_pack_name (&name, p->hash,"keep" ); +odb_pack_name (p->repo, &name, p->hash,"keep" );unlink_or_warn (name.buf); }strbuf_release (&name);
builtin/index-pack.c +2 -2
@@ -1479,7 +1479,7 @@ static void write_special_file(const char *suffix, const char *msg,if (pack_name) filename =derive_filename (pack_name,"pack" , suffix, &name_buf);else - filename =odb_pack_name (&name_buf, hash, suffix); + filename =odb_pack_name (the_repository, &name_buf, hash, suffix); fd =odb_pack_keep (filename);if (fd <0 ) {
@@ -1507,7 +1507,7 @@ static void rename_tmp_packfile(const char **final_name, {if (!*final_name ||strcmp (*final_name, curr_name)) {if (!*final_name) - *final_name =odb_pack_name (name, hash, ext); + *final_name =odb_pack_name (the_repository, name, hash, ext);if (finalize_object_file (curr_name, *final_name))die (_ ("unable to rename temporary '*.%s' file to '%s'" ), ext, *final_name);
builtin/pack-redundant.c +1 -1
@@ -690,7 +690,7 @@ int cmd_pack_redundant(int argc, const char **argv, const char *prefix UNUSED, s pl = red =pack_list_difference (local_packs, min);while (pl) {printf ("%s\n%s\n" , -odb_pack_name (&idx_name, pl->pack->hash,"idx" ), +odb_pack_name (pl->pack->repo, &idx_name, pl->pack->hash,"idx" ), pl->pack->pack_name); pl = pl->next; }
http.c +1 -1
@@ -2581,7 +2581,7 @@ struct http_pack_request *new_direct_http_pack_request( preq->url = url; -odb_pack_name (&preq->tmpfile, packed_git_hash,"pack" ); +odb_pack_name (the_repository, &preq->tmpfile, packed_git_hash,"pack" );strbuf_addstr (&preq->tmpfile,".temp" ); preq->packfile =fopen (preq->tmpfile.buf,"a" );if (!preq->packfile) {
packfile.c +4 -5
@@ -25,13 +25,12 @@#include "pack-revindex.h" #include "promisor-remote.h" -char *odb_pack_name (struct strbuf *buf, -const unsigned char *hash, -const char *ext) +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext) {strbuf_reset (buf); -strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (the_repository), -hash_to_hex (hash), ext); +strbuf_addf (buf,"%s/pack/pack-%s.%s" ,repo_get_object_directory (r), +hash_to_hex_algop (hash, r->hash_algo), ext);return buf->buf; }
packfile.h +2 -1
@@ -29,7 +29,8 @@ struct pack_entry {* * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" */ -char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); +char *odb_pack_name (struct repository *r,struct strbuf *buf, +const unsigned char *hash,const char *ext);/* * Return the basename of the packfile, omitting any containing directory
Signature · 1 line
2.47.1
PATCH v10 5/10 packfile: pass down repository to `has_object[_kept]_pack`
In reply to Karthik NayakThe functions has_object[_kept]_pack currently rely on the global variable the_repository. To eliminate global variable usage in packfile.c, we should progressively shift the dependency on the_repository to higher layers. Let's remove its usage from these functions and any related ones.
--- builtin/count-objects.c | 2 +- builtin/fsck.c | 2 +- builtin/pack-objects.c | 4 ++-- diff.c | 3 ++- list-objects.c | 3 ++- pack-bitmap.c | 2 +- packfile.c | 9 +++++---- packfile.h | 5 +++-- prune-packed.c | 2 +- reachable.c | 2 +- revision.c | 4 ++-- 11 files changed, 21 insertions(+), 17 deletions(-)
builtin/count-objects.c +1 -1
@@ -67,7 +67,7 @@ static int count_loose(const struct object_id *oid, const char *path,else { loose_size +=on_disk_bytes (st); loose++; -if (verbose &&has_object_pack (oid)) +if (verbose &&has_object_pack (the_repository, oid)) packed_loose++; }return 0 ;
builtin/fsck.c +1 -1
@@ -272,7 +272,7 @@ static void check_reachable_object(struct object *obj)if (!(obj->flags &HAS_OBJ )) {if (is_promisor_object (&obj->oid))return ; -if (has_object_pack (&obj->oid)) +if (has_object_pack (the_repository, &obj->oid))return ;/* it is in pack - forget about it */ printf_ln(_("missing %s %s"), printable_type(&obj->oid, obj->type),
builtin/pack-objects.c +2 -2
@@ -1529,7 +1529,7 @@ static int want_found_object(const struct object_id *oid, int exclude,return 0 ;if (ignore_packed_keep_in_core && p->pack_keep_in_core)return 0 ; -if (has_object_kept_pack (oid, flags)) +if (has_object_kept_pack (p->repo, oid, flags))return 0 ; }
@@ -3627,7 +3627,7 @@ static void show_cruft_commit(struct commit *commit, void *data) static int cruft_include_check_obj(struct object *obj, void *data UNUSED) { -return !has_object_kept_pack (&obj->oid,IN_CORE_KEEP_PACKS ); +return !has_object_kept_pack (to_pack.repo, &obj->oid,IN_CORE_KEEP_PACKS ); }static int cruft_include_check (struct commit *commit,void *data)
diff.c +2 -1
@@ -4041,7 +4041,8 @@ static int reuse_worktree_file(struct index_state *istate,* objects however would tend to be slower as they need * to be individually opened and inflated. */ -if (!FAST_WORKING_DIRECTORY && !want_file &&has_object_pack (oid)) +if (!FAST_WORKING_DIRECTORY && !want_file && +has_object_pack (istate->repo, oid))return 0 ;/*
list-objects.c +2 -1
@@ -41,7 +41,8 @@ static void show_object(struct traversal_context *ctx, {if (!ctx->show_object)return ; -if (ctx->revs->unpacked &&has_object_pack (&object->oid)) +if (ctx->revs->unpacked &&has_object_pack (ctx->revs->repo, + &object->oid))return ; ctx->show_object (object, name, ctx->show_data);
pack-bitmap.c +1 -1
@@ -1889,7 +1889,7 @@ static void filter_packed_objects_from_bitmap(struct bitmap_index *bitmap_git,bitmap_unset (result, i);for (i =0 ; i < eindex->count; ++i) { -if (has_object_pack (&eindex->objects[i]->oid)) +if (has_object_pack (the_repository, &eindex->objects[i]->oid))bitmap_unset (result, objects_nr + i); } }
packfile.c +5 -4
@@ -2143,16 +2143,17 @@ int find_kept_pack_entry(struct repository *r,return 0 ; } -int has_object_pack (const struct object_id *oid) +int has_object_pack (struct repository *r,const struct object_id *oid) {struct pack_entry e; -return find_pack_entry (the_repository, oid, &e); +return find_pack_entry (r, oid, &e); } -int has_object_kept_pack (const struct object_id *oid,unsigned flags) +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags) {struct pack_entry e; -return find_kept_pack_entry (the_repository, oid, flags, &e); +return find_kept_pack_entry (r, oid, flags, &e); } int for_each_object_in_pack(struct packed_git *p,
packfile.h +3 -2
@@ -193,8 +193,9 @@ const struct packed_git *has_packed_and_bad(struct repository *, const struct obint find_pack_entry (struct repository *r,const struct object_id *oid,struct pack_entry *e);int find_kept_pack_entry (struct repository *r,const struct object_id *oid,unsigned flags,struct pack_entry *e); -int has_object_pack (const struct object_id *oid); -int has_object_kept_pack (const struct object_id *oid,unsigned flags); +int has_object_pack (struct repository *r,const struct object_id *oid); +int has_object_kept_pack (struct repository *r,const struct object_id *oid, +unsigned flags);/* * Return 1 if an object in a promisor packfile is or refers to the given
prune-packed.c +1 -1
@@ -24,7 +24,7 @@ static int prune_object(const struct object_id *oid, const char *path, {int *opts = data; -if (!has_object_pack (oid)) +if (!has_object_pack (the_repository, oid))return 0 ;if (*opts &PRUNE_PACKED_DRY_RUN )
reachable.c +1 -1
@@ -239,7 +239,7 @@ static int want_recent_object(struct recent_data *data, const struct object_id *oid) {if (data->ignore_in_core_kept_packs && -has_object_kept_pack (oid,IN_CORE_KEEP_PACKS )) +has_object_kept_pack (data->revs->repo, oid,IN_CORE_KEEP_PACKS ))return 0 ;return 1 ; }
revision.c +2 -2
@@ -4103,10 +4103,10 @@ enum commit_action get_commit_action(struct rev_info *revs, struct commit *commi {if (commit->object.flags &SHOWN )return commit_ignore; -if (revs->unpacked &&has_object_pack (&commit->object.oid)) +if (revs->unpacked &&has_object_pack (revs->repo, &commit->object.oid))return commit_ignore;if (revs->no_kept_objects) { -if (has_object_kept_pack (&commit->object.oid, +if (has_object_kept_pack (revs->repo, &commit->object.oid, revs->keep_pack_cache_flags))return commit_ignore; }
Signature · 1 line
2.47.1