An attempt to move packfile funcs to its own file
cover An attempt to move packfile funcs to its own file
While investigating annotating packfiles and loose objects to support connectivity checks in partial clones [1], I decided to make the effort to separate packfile-related code from sha1_file.c into its own file, to make it easier to both code such changes and review them. Here is the beginning of those efforts.
Is this something worth doing, and if yes, is this in the right direction?
[1] https://public-inbox.org/git/[email protected]/
Jonathan Tan (10):
pack: move pack name-related functions
pack: move static state variables
pack: move pack_report()
pack: move open_pack_index(), parse_pack_index()
pack: move release_pack_memory()
pack: move pack-closing functions
pack: move use_pack()
pack: move unuse_pack()
pack: move add_packed_git()
pack: move install_packed_git()
Makefile | 1 + builtin/am.c | 1 + builtin/clone.c | 1 + builtin/count-objects.c | 1 + builtin/fetch.c | 1 + builtin/merge.c | 1 + builtin/pack-redundant.c | 1 + cache.h | 45 ---- connected.c | 1 + git-compat-util.h | 2 - pack.c | 669 +++++++++++++++++++++++++++++++++++++++++++++++ pack.h | 51 ++++ sha1_file.c | 667 ---------------------------------------------- sha1_name.c | 1 + streaming.c | 1 + 15 files changed, 730 insertions(+), 714 deletions(-) create mode 100644 pack.c
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
RFC PATCH 1/10 pack: move pack name-related functions
Currently, sha1_file.c and cache.h contain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.
Create a new file, pack.c, to hold all packfile-related functions currently in sha1_file.c, and designate pack.h to hold these packfile-related functions.
In this commit, the pack name-related functions are moved. Subsequent commits will move the other functions.
--- Makefile | 1 + builtin/pack-redundant.c | 1 + cache.h | 23 ----------------------- pack.c | 23 +++++++++++++++++++++++ pack.h | 23 +++++++++++++++++++++++ sha1_file.c | 22 ---------------------- 6 files changed, 48 insertions(+), 45 deletions(-) create mode 100644 pack.c
Makefile +1 -0
@@ -816,6 +816,7 @@ LIB_OBJS += notes-merge.o LIB_OBJS += notes-utils.o LIB_OBJS += object.o LIB_OBJS += oidset.o +LIB_OBJS += pack.o LIB_OBJS += pack-bitmap.o LIB_OBJS += pack-bitmap-write.o LIB_OBJS += pack-check.o
builtin/pack-redundant.c +1 -0
@@ -7,6 +7,7 @@*/ #include "builtin.h" +#include "pack.h" #define BLKSIZE 512
cache.h +0 -23
@@ -902,20 +902,6 @@ extern void check_repository_format(void);*/ extern const char *sha1_file_name (const unsigned char *sha1); -/* -* Return the name of the (local) packfile with the specified sha1 in -* its name. The return value is a pointer to memory that is -* overwritten each time this function is called. -*/ -extern char *sha1_pack_name (const unsigned char *sha1); - -/* -* Return the name of the (local) pack index file with the specified -* sha1 in its name. The return value is a pointer to memory that is -* overwritten each time this function is called. -*/ -extern char *sha1_pack_index_name (const unsigned char *sha1); -/* * Return an abbreviated sha1 unique within this repository's object database. * The result will be at least `len` characters long, and will be NUL
@@ -1648,15 +1634,6 @@ extern void pack_report(void);*/ extern int odb_mkstemp (struct strbuf *template,const char *pattern); -/* -* Generate the filename to be used for a pack file with checksum "sha1" and -* extension "ext". The result is written into the strbuf "buf", overwriting -* any existing contents. A pointer to buf->buf is returned as a convenience. -* -* Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" -*/ -extern char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); -/* * Create a pack .keep file named "name" (which should generally be the output * of odb_pack_name). Returns a file descriptor opened for writing, or -1 on
pack.c new file +23 -0
@@ -0,0 +1,23 @@ +#include "cache.h" + +char *odb_pack_name (struct strbuf *buf, +const unsigned char *sha1, +const char *ext) +{ +strbuf_reset (buf); +strbuf_addf (buf,"%s/pack/pack-%s.%s" ,get_object_directory (), +sha1_to_hex (sha1), ext); +return buf->buf; +} + +char *sha1_pack_name (const unsigned char *sha1) +{ +static struct strbuf buf =STRBUF_INIT ; +return odb_pack_name (&buf, sha1,"pack" ); +} + +char *sha1_pack_index_name (const unsigned char *sha1) +{ +static struct strbuf buf =STRBUF_INIT ; +return odb_pack_name (&buf, sha1,"idx" ); +}
pack.h +23 -0
@@ -101,4 +101,27 @@ extern int read_pack_header(int fd, struct pack_header *);extern struct sha1file *create_tmp_packfile (char **pack_tmp_name);extern void finish_tmp_packfile (struct strbuf *name_buffer,const char *pack_tmp_name,struct pack_idx_entry **written_list,uint32_t nr_written,struct pack_idx_option *pack_idx_opts,unsigned char sha1[]); +/* +* Generate the filename to be used for a pack file with checksum "sha1" and +* extension "ext". The result is written into the strbuf "buf", overwriting +* any existing contents. A pointer to buf->buf is returned as a convenience. +* +* Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" +*/ +extern char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); + +/* +* Return the name of the (local) packfile with the specified sha1 in +* its name. The return value is a pointer to memory that is +* overwritten each time this function is called. +*/ +extern char *sha1_pack_name (const unsigned char *sha1); + +/* +* Return the name of the (local) pack index file with the specified +* sha1 in its name. The return value is a pointer to memory that is +* overwritten each time this function is called. +*/ +extern char *sha1_pack_index_name (const unsigned char *sha1); +#endif
sha1_file.c +0 -22
@@ -278,28 +278,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,return buf->buf; } -char *odb_pack_name (struct strbuf *buf, -const unsigned char *sha1, -const char *ext) -{ -strbuf_reset (buf); -strbuf_addf (buf,"%s/pack/pack-%s.%s" ,get_object_directory (), -sha1_to_hex (sha1), ext); -return buf->buf; -} - -char *sha1_pack_name (const unsigned char *sha1) -{ -static struct strbuf buf =STRBUF_INIT ; -return odb_pack_name (&buf, sha1,"pack" ); -} - -char *sha1_pack_index_name (const unsigned char *sha1) -{ -static struct strbuf buf =STRBUF_INIT ; -return odb_pack_name (&buf, sha1,"idx" ); -} -struct alternate_object_database *alt_odb_list;static struct alternate_object_database **alt_odb_tail;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
Re: [RFC PATCH 01/10] pack: move pack name-related functions
Quoted 7 lines · Jonathan Tan
Currently,
sha1_file.candcache.hcontain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.Create a new file,
pack.c, to hold all packfile-related functions currently insha1_file.c, and designatepack.hto hold these packfile-related functions.
There are also packed refs, so one could (like I did) think that pack.c is for generic packing of things, maybe packfile.c would be more clear?
Re: [RFC PATCH 01/10] pack: move pack name-related functions
Quoted 5 lines · On Tue, 8 Aug 2017 13:36:24 -0700 Stefan Beller <[email protected]> wrote:
Quoted 4 lines · Jonathan Tan
Currently,
sha1_file.candcache.hcontain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.
Quoted 7 lines
Quoted 3 lines
Create a new file,
pack.c, to hold all packfile-related functions currently insha1_file.c, and designatepack.hto hold these packfile-related functions.There are also packed refs, so one could (like I did) think that
pack.cis for generic packing of things, maybepackfile.cwould be more clear?
Good point. I'll use packfile.c and packfile.h in the next version.
Re: [RFC PATCH 01/10] pack: move pack name-related functions
Quoted 8 lines · Jonathan Tan
Quoted 4 lines · On Tue, 8 Aug 2017 13:36:24 -0700 Stefan Beller <[email protected]> wrote:
There are also packed refs, so one could (like I did) think that
pack.cis for generic packing of things, maybepackfile.cwould be more clear?Good point. I'll use
packfile.candpackfile.hin the next version.
It looks like you used "packfile.c" and "pack.h" in v2. Is there a reason why it's not using "packfile.h"?
Re: [RFC PATCH 01/10] pack: move pack name-related functions
Quoted 4 lines · On Wed, 9 Aug 2017 14:00:40 +0200 Christian Couder…
Quoted 3 lines · Jonathan Tan
Quoted 1 line · On Tue, 8 Aug 2017 13:36:24 -0700 Stefan Beller <[email protected]> wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
There are also packed refs, so one could (like I did) think that
pack.cis for generic packing of things, maybepackfile.cwould be more clear?Good point. I'll use
packfile.candpackfile.hin the next version.It looks like you used "
packfile.c" and "pack.h" in v2. Is there a reason why it's not using "packfile.h"?
Ah, I forgot to mention this in the cover letter. I thought that one header was sufficient to cover all pack-related things, so if we wanted to know which files used pack-related things, we would only need to search for one string instead of two. Also, the division between "pack.h" and the hypothetical "packfile.h" was not so clear to me.
Re: [RFC PATCH 01/10] pack: move pack name-related functions
Quoted 13 lines · Jonathan Tan
Quoted 10 lines · On Wed, 9 Aug 2017 14:00:40 +0200 Christian Couder…
Quoted 8 lines · Jonathan Tan
Quoted 4 lines · On Tue, 8 Aug 2017 13:36:24 -0700 Stefan Beller <[email protected]> wrote:
There are also packed refs, so one could (like I did) think that
pack.cis for generic packing of things, maybepackfile.cwould be more clear?Good point. I'll use
packfile.candpackfile.hin the next version.
Quoted 9 lines
Quoted 2 lines
It looks like you used "
packfile.c" and "pack.h" in v2. Is there a reason why it's not using "packfile.h"?Ah, I forgot to mention this in the cover letter. I thought that one header was sufficient to cover all pack-related things, so if we wanted to know which files used pack-related things, we would only need to search for one string instead of two. Also, the division between "
pack.h" and the hypothetical "packfile.h" was not so clear to me.
I prefer having source and the header files that export the functions have matching names to make it easy to find them. I would prefer packfile.h vs pack.h myself.
Re: [RFC PATCH 01/10] pack: move pack name-related functions
Quoted 11 lines · Ben Peart
Quoted 5 lines · Jonathan Tan
Ah, I forgot to mention this in the cover letter. I thought that one header was sufficient to cover all pack-related things, so if we wanted to know which files used pack-related things, we would only need to search for one string instead of two. Also, the division between "
pack.h" and the hypothetical "packfile.h" was not so clear to me.I prefer having source and the header files that export the functions have matching names to make it easy to find them. I would prefer
packfile.hvspack.hmyself.
Meaning "If we have packfile.c, packfile.h is preferrable over pack.h"? I tend to agree with that.
Re: [RFC PATCH 01/10] pack: move pack name-related functions
Quoted 10 lines · On Fri, 11 Aug 2017 14:34:27 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 8 lines · Ben Peart
Quoted 5 lines · Jonathan Tan
Ah, I forgot to mention this in the cover letter. I thought that one header was sufficient to cover all pack-related things, so if we wanted to know which files used pack-related things, we would only need to search for one string instead of two. Also, the division between "
pack.h" and the hypothetical "packfile.h" was not so clear to me.
Quoted 6 lines
Quoted 3 lines
I prefer having source and the header files that export the functions have matching names to make it easy to find them. I would prefer
packfile.hvspack.hmyself.Meaning "If we have
packfile.c,packfile.his preferrable overpack.h"? I tend to agree with that.
Fair enough - I've changed it so that the functions now go into packfile.h. I'll send it out once I know what to base it on (at least jt/sha1-file-cleanup, and a few more branches that also modify sha1_file.c).
RFC PATCH 2/10 pack: move static state variables
In reply to Jonathan Tansha1_file.c declares some static variables that store packfile-related state. Move them to pack.c.
They are temporarily made global, but subsequent commits will restore their scope back to static.
--- pack.c | 14 ++++++++++++++ pack.h | 9 +++++++++ sha1_file.c | 13 ------------- 3 files changed, 23 insertions(+), 13 deletions(-)
pack.c +14 -0
@@ -1,4 +1,5 @@#include "cache.h" +#include "mru.h" char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1,
@@ -21,3 +22,16 @@ char *sha1_pack_index_name(const unsigned char *sha1)static struct strbuf buf =STRBUF_INIT ;return odb_pack_name (&buf, sha1,"idx" ); } + +unsigned int pack_used_ctr; +unsigned int pack_mmap_calls; +unsigned int peak_pack_open_windows; +unsigned int pack_open_windows; +unsigned int pack_open_fds; +unsigned int pack_max_fds; +size_t peak_pack_mapped; +size_t pack_mapped; +struct packed_git *packed_git; + +static struct mru packed_git_mru_storage; +struct mru *packed_git_mru = &packed_git_mru_storage;
pack.h +9 -0
@@ -124,4 +124,13 @@ extern char *sha1_pack_name(const unsigned char *sha1);*/ extern char *sha1_pack_index_name (const unsigned char *sha1); +extern unsigned int pack_used_ctr; +extern unsigned int pack_mmap_calls; +extern unsigned int peak_pack_open_windows; +extern unsigned int pack_open_windows; +extern unsigned int pack_open_fds; +extern unsigned int pack_max_fds; +extern size_t peak_pack_mapped; +extern size_t pack_mapped; +#endif
sha1_file.c +0 -13
@@ -682,19 +682,6 @@ static int has_loose_object(const unsigned char *sha1)return check_and_freshen (sha1,0 ); } -static unsigned int pack_used_ctr; -static unsigned int pack_mmap_calls; -static unsigned int peak_pack_open_windows; -static unsigned int pack_open_windows; -static unsigned int pack_open_fds; -static unsigned int pack_max_fds; -static size_t peak_pack_mapped; -static size_t pack_mapped; -struct packed_git *packed_git; - -static struct mru packed_git_mru_storage; -struct mru *packed_git_mru = &packed_git_mru_storage; -void pack_report (void ) { fprintf(stderr,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
RFC PATCH 3/10 pack: move pack_report()
In reply to Jonathan Tan--- cache.h | 2 -- pack.c | 24 ++++++++++++++++++++++++ pack.h | 2 ++ sha1_file.c | 24 ------------------------ 4 files changed, 26 insertions(+), 26 deletions(-)
cache.h +0 -2
@@ -1624,8 +1624,6 @@ unsigned long approximate_object_count(void);extern struct packed_git *find_sha1_pack (const unsigned char *sha1,struct packed_git *packs); -extern void pack_report (void ); -/* * Create a temporary file rooted in the object database directory, or * die on failure. The filename is taken from "pattern", which should have the
pack.c +24 -0
@@ -35,3 +35,27 @@ struct packed_git *packed_git;static struct mru packed_git_mru_storage;struct mru *packed_git_mru = &packed_git_mru_storage; + +#define SZ_FMT PRIuMAX +static inline uintmax_t sz_fmt (size_t s) {return s; } + +void pack_report (void ) +{ +fprintf (stderr, +"pack_report: getpagesize() = %10" SZ_FMT "\n" +"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" +"pack_report: core.packedGitLimit = %10" SZ_FMT "\n" , +sz_fmt (getpagesize ()), +sz_fmt (packed_git_window_size), +sz_fmt (packed_git_limit)); +fprintf (stderr, +"pack_report: pack_used_ctr = %10u\n" +"pack_report: pack_mmap_calls = %10u\n" +"pack_report: pack_open_windows = %10u / %10u\n" +"pack_report: pack_mapped = " +"%10" SZ_FMT " / %10" SZ_FMT "\n" , + pack_used_ctr, + pack_mmap_calls, + pack_open_windows, peak_pack_open_windows, +sz_fmt (pack_mapped),sz_fmt (peak_pack_mapped)); +}
pack.h +2 -0
@@ -133,4 +133,6 @@ extern unsigned int pack_max_fds;extern size_t peak_pack_mapped;extern size_t pack_mapped; +extern void pack_report (void ); +#endif
sha1_file.c +0 -24
@@ -29,9 +29,6 @@#include "mergesort.h" #include "quote.h" -#define SZ_FMT PRIuMAX -static inline uintmax_t sz_fmt (size_t s) {return s; } -const unsigned char null_sha1[20 ];const struct object_id null_oid;const struct object_id empty_tree_oid = {
@@ -682,27 +679,6 @@ static int has_loose_object(const unsigned char *sha1)return check_and_freshen (sha1,0 ); } -void pack_report (void ) -{ -fprintf (stderr, -"pack_report: getpagesize() = %10" SZ_FMT "\n" -"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" -"pack_report: core.packedGitLimit = %10" SZ_FMT "\n" , -sz_fmt (getpagesize ()), -sz_fmt (packed_git_window_size), -sz_fmt (packed_git_limit)); -fprintf (stderr, -"pack_report: pack_used_ctr = %10u\n" -"pack_report: pack_mmap_calls = %10u\n" -"pack_report: pack_open_windows = %10u / %10u\n" -"pack_report: pack_mapped = " -"%10" SZ_FMT " / %10" SZ_FMT "\n" , - pack_used_ctr, - pack_mmap_calls, - pack_open_windows, peak_pack_open_windows, -sz_fmt (pack_mapped),sz_fmt (peak_pack_mapped)); -} -/* * Open and mmap the index file at path, perform a couple of * consistency checks, then record its information to p. Return 0 on
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
RFC PATCH 4/10 pack: move open_pack_index(), parse_pack_index()
In reply to Jonathan Tan--- builtin/count-objects.c | 1 + cache.h | 8 --- pack.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++ pack.h | 8 +++ sha1_file.c | 140 --------------------------------------------- sha1_name.c | 1 + 6 files changed, 159 insertions(+), 148 deletions(-)
builtin/count-objects.c +1 -0
@@ -10,6 +10,7 @@#include "builtin.h" #include "parse-options.h" #include "quote.h" +#include "pack.h" static unsigned long garbage;static off_t size_garbage;
cache.h +0 -8
@@ -1603,8 +1603,6 @@ struct pack_entry {struct packed_git *p; }; -extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); -/* A hook to report invalid files in pack directory */ #define PACKDIR_FILE_PACK 1#define PACKDIR_FILE_IDX 2
@@ -1639,12 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* mmap the index file for the specified packfile (if it is not -* already mmapped). Return 0 on success. -*/ -extern int open_pack_index (struct packed_git *); -/* * munmap the index file for the specified packfile (if it is * currently mmapped).
pack.c +149 -0
157 lines not on this page · raw message
pack.h +8 -0
@@ -135,4 +135,12 @@ extern size_t pack_mapped;extern void pack_report (void ); +/* +* mmap the index file for the specified packfile (if it is not +* already mmapped). Return 0 on success. +*/ +extern int open_pack_index (struct packed_git *); + +extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +#endif
sha1_file.c +0 -140
152 lines not on this page · raw message
sha1_name.c +1 -0
@@ -9,6 +9,7 @@#include "remote.h" #include "dir.h" #include "sha1-array.h" +#include "pack.h" static int get_sha1_oneline (const char *,unsigned char *,struct commit_list *);
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
Re: [RFC PATCH 04/10] pack: move open_pack_index(), parse_pack_index()
Quoted 9 lines · Jonathan Tan
Signed-off-by Jonathan Tan <[email protected]>--- builtin/count-objects.c | 1 + cache.h | 8 --- pack.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++ pack.h | 8 +++ sha1_file.c | 140 --------------------------------------------- sha1_name.c | 1 + 6 files changed, 159 insertions(+), 148 deletions(-)
This patch is a bit strange...
Quoted 18 lines
pack.c +13
-0+static struct packed_git *alloc_packed_git (int extra) +{ +struct packed_git *p =xmalloc (st_add (sizeof (*p), extra)); +memset (p,0 ,sizeof (*p)); + p->pack_fd =-1 ; +return p; +} + +struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) +{ +const char *path =sha1_pack_name (sha1); +size_t alloc =st_add (strlen (path),1 ); +struct packed_git *p =alloc_packed_git (alloc);
pack.c void pack_report(void) · line 210
Quoted 10 lines
pack.c +10
-0+ +memcpy (p->pack_name, path, alloc);/* includes NUL */ +hashcpy (p->sha1, sha1); +if (check_packed_git_idx (idx_path, p)) { +free (p); +return NULL ; + } + +return p; +}
We see these two functions appear in pack.c
Quoted 15 lines
sha1_file.c +0
-6@@ -1300,22 +1176,6 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)return p; } -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) -{ -const char *path =sha1_pack_name (sha1); -size_t alloc =st_add (strlen (path),1 ); -struct packed_git *p =alloc_packed_git (alloc); -
sha1_file.c struct packed_git *add_packed_git(const char *path, size_t path_len, int local) · line 1318
Quoted 10 lines
sha1_file.c +0
-10-memcpy (p->pack_name, path, alloc);/* includes NUL */ -hashcpy (p->sha1, sha1); -if (check_packed_git_idx (idx_path, p)) { -free (p); -return NULL ; - } - -return p; -} -
And we see parse_pack_index() came from sha1_file.c
But where did alloc_packed_git() come from? Was the patch split incorrectly or something?
When I applied the whole series and did
git blame -s -w -M -C -C master.. pack.c
expecting that pretty much everything has come from sha1_file.c but noticed that some lines were actually blamed to a version of pack.c and these functions were among them.
Re: [RFC PATCH 04/10] pack: move open_pack_index(), parse_pack_index()
Quoted 76 lines · On Tue, 08 Aug 2017 13:19:23 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 9 lines · Jonathan Tan
Signed-off-by Jonathan Tan <[email protected]>--- builtin/count-objects.c | 1 + cache.h | 8 --- pack.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++ pack.h | 8 +++ sha1_file.c | 140 --------------------------------------------- sha1_name.c | 1 + 6 files changed, 159 insertions(+), 148 deletions(-)This patch is a bit strange...
Quoted 18 lines
pack.c +13
-0+static struct packed_git *alloc_packed_git (int extra) +{ +struct packed_git *p =xmalloc (st_add (sizeof (*p), extra)); +memset (p,0 ,sizeof (*p)); + p->pack_fd =-1 ; +return p; +} + +struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) +{ +const char *path =sha1_pack_name (sha1); +size_t alloc =st_add (strlen (path),1 ); +struct packed_git *p =alloc_packed_git (alloc);pack.c void pack_report(void) · line 210
Quoted 10 lines
pack.c +10
-0+ +memcpy (p->pack_name, path, alloc);/* includes NUL */ +hashcpy (p->sha1, sha1); +if (check_packed_git_idx (idx_path, p)) { +free (p); +return NULL ; + } + +return p; +}We see these two functions appear in
pack.cQuoted 15 lines
sha1_file.c +0
-6@@ -1300,22 +1176,6 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)return p; } -struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path) -{ -const char *path =sha1_pack_name (sha1); -size_t alloc =st_add (strlen (path),1 ); -struct packed_git *p =alloc_packed_git (alloc); -sha1_file.c struct packed_git *add_packed_git(const char *path, size_t path_len, int local) · line 1318
Quoted 10 lines
sha1_file.c +0
-10-memcpy (p->pack_name, path, alloc);/* includes NUL */ -hashcpy (p->sha1, sha1); -if (check_packed_git_idx (idx_path, p)) { -free (p); -return NULL ; - } - -return p; -} -And we see
parse_pack_index()came fromsha1_file.cBut where did
alloc_packed_git()come from? Was the patch split incorrectly or something?
Quoted 7 lines
When I applied the whole series and did
git blame -s -w -M -C -C master..
pack.cexpecting that pretty much everything has come from
sha1_file.cbut noticed that some lines were actually blamed to a version ofpack.cand these functions were among them.
alloc_packed_git() in pack.c is a duplicate of the function of the same name in sha1_file.c in this patch, because at this patch, there are still functions in both files using this function. A subsequent patch in this patch set will remove it from pack.c.
I'll add a note explaining this to this patch in the next version.
RFC PATCH 5/10 pack: move release_pack_memory()
In reply to Jonathan TanThe function unuse_one_window() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.
--- git-compat-util.h | 2 -- pack.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ pack.h | 4 ++++ sha1_file.c | 49 ------------------------------------------------- 4 files changed, 53 insertions(+), 51 deletions(-)
git-compat-util.h +0 -2
@@ -749,8 +749,6 @@ const char *inet_ntop(int af, const void *src, char *dst, size_t size);extern int git_atexit (void (*handler)(void ));#endif -extern void release_pack_memory (size_t ); -typedef void (*try_to_free_t )(size_t );extern try_to_free_t set_try_to_free_routine (try_to_free_t );
pack.c +49 -0
52 lines not on this page · raw message
pack.h +4 -0
@@ -143,4 +143,8 @@ extern int open_pack_index(struct packed_git *);extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +extern int unuse_one_window (struct packed_git *current); + +extern void release_pack_memory (size_t ); +#endif
sha1_file.c +0 -49
55 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
RFC PATCH 6/10 pack: move pack-closing functions
In reply to Jonathan TanThe function close_pack_fd() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.
---
In doing this, I discovered that some builtins close the packs even though they, in theory, should not know anything about how objects are stored. Can we remove those calls? (The tests pass with those calls removed.)
--- builtin/am.c | 1 + builtin/clone.c | 1 + builtin/fetch.c | 1 + builtin/merge.c | 1 + cache.h | 8 -------- pack.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ pack.h | 9 +++++++++ sha1_file.c | 55 ------------------------------------------------------- 8 files changed, 67 insertions(+), 63 deletions(-)
builtin/am.c +1 -0
@@ -31,6 +31,7 @@#include "mailinfo.h" #include "apply.h" #include "string-list.h" +#include "pack.h" /** * Returns 1 if the file is empty or does not exist, 0 otherwise.
builtin/clone.c +1 -0
@@ -25,6 +25,7 @@#include "remote.h" #include "run-command.h" #include "connected.h" +#include "pack.h" /* * Overall FIXMEs:
builtin/fetch.c +1 -0
@@ -17,6 +17,7 @@#include "connected.h" #include "argv-array.h" #include "utf8.h" +#include "pack.h" static const char *const builtin_fetch_usage[] = {N_ ("git fetch [<options>] [<repository> [<refspec>...]]" ),
builtin/merge.c +1 -0
@@ -32,6 +32,7 @@#include "gpg-interface.h" #include "sequencer.h" #include "string-list.h" +#include "pack.h" #define DEFAULT_TWOHEAD (1<<0)#define DEFAULT_OCTOPUS (1<<1)
cache.h +0 -8
@@ -1637,15 +1637,7 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* munmap the index file for the specified packfile (if it is -* currently mmapped). -*/ -extern void close_pack_index (struct packed_git *); -extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *); -extern void close_pack_windows (struct packed_git *); -extern void close_all_packs (void );extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
pack.c +54 -0
57 lines not on this page · raw message
pack.h +9 -0
@@ -147,4 +147,13 @@ extern int unuse_one_window(struct packed_git *current);extern void release_pack_memory (size_t ); +extern void close_pack_windows (struct packed_git *); +extern int close_pack_fd (struct packed_git *); +/* +* munmap the index file for the specified packfile (if it is +* currently mmapped). +*/ +extern void close_pack_index (struct packed_git *); +extern void close_all_packs (void ); +#endif
sha1_file.c +0 -55
67 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
RFC PATCH 7/10 pack: move use_pack()
In reply to Jonathan Tanopen_packed_git is made global.
---
Unlike the other commits where variables and functions are made global and then remade static, open_packed_git is not remade static (because a function in sha1_file.c still uses it).
--- cache.h | 1 - pack.c | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- pack.h | 14 +-- sha1_file.c | 285 -------------------------------------------------------- streaming.c | 1 + 5 files changed, 299 insertions(+), 305 deletions(-)
cache.h +0 -1
@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
pack.c +294 -9
325 lines not on this page · raw message
pack.h +4 -10
@@ -124,14 +124,7 @@ extern char *sha1_pack_name(const unsigned char *sha1);*/ extern char *sha1_pack_index_name (const unsigned char *sha1); -extern unsigned int pack_used_ctr; -extern unsigned int pack_mmap_calls; -extern unsigned int peak_pack_open_windows; -extern unsigned int pack_open_windows;extern unsigned int pack_open_fds; -extern unsigned int pack_max_fds; -extern size_t peak_pack_mapped; -extern size_t pack_mapped;extern void pack_report (void );
@@ -143,12 +136,9 @@ extern int open_pack_index(struct packed_git *);extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); -extern int unuse_one_window (struct packed_git *current); -extern void release_pack_memory (size_t );extern void close_pack_windows (struct packed_git *); -extern int close_pack_fd (struct packed_git *);/* * munmap the index file for the specified packfile (if it is * currently mmapped).
@@ -156,4 +146,8 @@ extern int close_pack_fd(struct packed_git *);extern void close_pack_index (struct packed_git *);extern void close_all_packs (void ); +extern int open_packed_git (struct packed_git *p); + +extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *); +#endif
sha1_file.c +0 -285
297 lines not on this page · raw message
streaming.c +1 -0
@@ -3,6 +3,7 @@*/ #include "cache.h" #include "streaming.h" +#include "pack.h" enum input_source { stream_error =-1 ,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
RFC PATCH 8/10 pack: move unuse_pack()
In reply to Jonathan Tan--- cache.h | 1 - pack.c | 9 +++++++++ pack.h | 1 + sha1_file.c | 9 --------- 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -1
@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
pack.c +9 -0
@@ -596,3 +596,12 @@ unsigned char *use_pack(struct packed_git *p, *left = win->len -xsize_t (offset);return win->base + offset; } + +void unuse_pack (struct pack_window **w_cursor) +{ +struct pack_window *w = *w_cursor; +if (w) { + w->inuse_cnt--; + *w_cursor =NULL ; + } +}
pack.h +1 -0
@@ -149,5 +149,6 @@ extern void close_all_packs(void);extern int open_packed_git (struct packed_git *p);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *); +extern void unuse_pack (struct pack_window **);#endif
sha1_file.c +0 -9
@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,return ret; } -void unuse_pack (struct pack_window **w_cursor) -{ -struct pack_window *w = *w_cursor; -if (w) { - w->inuse_cnt--; - *w_cursor =NULL ; - } -} -static struct packed_git *alloc_packed_git (int extra) {struct packed_git *p =xmalloc (st_add (sizeof (*p), extra));
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
RFC PATCH 9/10 pack: move add_packed_git()
In reply to Jonathan Tan--- cache.h | 1 - connected.c | 1 + pack.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ pack.h | 1 + sha1_file.c | 61 ------------------------------------------------------------- 5 files changed, 55 insertions(+), 62 deletions(-)
cache.h +0 -1
@@ -1638,7 +1638,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);extern int odb_pack_keep (const char *name);extern void clear_delta_base_cache (void ); -extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);/* * Make sure that a pointer access into an mmap'd index file is within bounds,
connected.c +1 -0
@@ -3,6 +3,7 @@#include "sigchain.h" #include "connected.h" #include "transport.h" +#include "pack.h" /* * If we feed all the commits we want to verify to this command
pack.c +53 -0
56 lines not on this page · raw message
pack.h +1 -0
@@ -150,5 +150,6 @@ extern int open_packed_git(struct packed_git *p);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void unuse_pack (struct pack_window **); +extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);#endif
sha1_file.c +0 -61
67 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
RFC PATCH 10/10 pack: move install_packed_git()
In reply to Jonathan Tan--- cache.h | 1 - pack.c | 11 ++++++++++- pack.h | 4 ++-- sha1_file.c | 9 --------- 4 files changed, 12 insertions(+), 13 deletions(-)
cache.h +0 -1
@@ -1611,7 +1611,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);extern void prepare_packed_git (void );extern void reprepare_packed_git (void ); -extern void install_packed_git (struct packed_git *pack);/* * Give a rough count of objects in the repository. This sacrifices accuracy
pack.c +10 -1
@@ -28,7 +28,7 @@ static unsigned int pack_used_ctr;static unsigned int pack_mmap_calls;static unsigned int peak_pack_open_windows;static unsigned int pack_open_windows; -unsigned int pack_open_fds; +static unsigned int pack_open_fds;static unsigned int pack_max_fds;static size_t peak_pack_mapped;static size_t pack_mapped;
@@ -658,3 +658,12 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)hashclr (p->sha1);return p; } + +void install_packed_git (struct packed_git *pack) +{ +if (pack->pack_fd !=-1 ) + pack_open_fds++; + + pack->next = packed_git; + packed_git = pack; +}
pack.h +2 -2
@@ -124,8 +124,6 @@ extern char *sha1_pack_name(const unsigned char *sha1);*/ extern char *sha1_pack_index_name (const unsigned char *sha1); -extern unsigned int pack_open_fds; -extern void pack_report (void );/*
@@ -152,4 +150,6 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_textern void unuse_pack (struct pack_window **);extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +extern void install_packed_git (struct packed_git *pack); +#endif
sha1_file.c +0 -9
@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,return ret; } -void install_packed_git (struct packed_git *pack) -{ -if (pack->pack_fd !=-1 ) - pack_open_fds++; - - pack->next = packed_git; - packed_git = pack; -} -void (*report_garbage)(unsigned seen_bits,const char *path); static void report_helper(const struct string_list *list,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
Quoted 8 lines · Jonathan Tan
While investigating annotating packfiles and loose objects to support connectivity checks in partial clones [1], I decided to make the effort to separate packfile-related code from
sha1_file.cinto its own file, to make it easier to both code such changes and review them. Here is the beginning of those efforts.Is this something worth doing, and if yes, is this in the right direction?
Overall I think it is a good idea to slim down sha1_file.c *if* we can keep the exposed surface area small enough.
I wonder if the names "pack.[ch]" communicate well that these are "object access layer that is about reading from packfiles". The writer side is called "pack-objects.[ch]".
This may have to make some symbols that used to be private to the "object access" layer, which was what sha1_file.c was about, global symbols. After moving things around, do we end up exposing too many implementation details to the world outside the "object access" layer? I'd assume they are limited to the resulting pack.h and it would be OK as long as nobody other than sha1_file.c and pack.c would inculde it, though.
Thanks.
Quoted 32 lines
[1] https://public-inbox.org/git/[email protected]/
Jonathan Tan (10):
pack: move pack name-related functions
pack: move static state variables
pack: movepack_report()
pack: moveopen_pack_index(),parse_pack_index()
pack: moverelease_pack_memory()
pack: move pack-closing functions
pack: moveuse_pack()
pack: moveunuse_pack()
pack: moveadd_packed_git()
pack: moveinstall_packed_git()Makefile | 1 + builtin/am.c | 1 + builtin/clone.c | 1 + builtin/count-objects.c | 1 + builtin/fetch.c | 1 + builtin/merge.c | 1 + builtin/pack-redundant.c | 1 + cache.h | 45 ---- connected.c | 1 + git-compat-util.h | 2 - pack.c | 669 +++++++++++++++++++++++++++++++++++++++++++++++ pack.h | 51 ++++ sha1_file.c | 667 ---------------------------------------------- sha1_name.c | 1 + streaming.c | 1 + 15 files changed, 730 insertions(+), 714 deletions(-) create mode 100644 pack.c
Quoted 8 lines · On Tue, 08 Aug 2017 13:05:05 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 6 lines · Jonathan Tan
While investigating annotating packfiles and loose objects to support connectivity checks in partial clones [1], I decided to make the effort to separate packfile-related code from
sha1_file.cinto its own file, to make it easier to both code such changes and review them. Here is the beginning of those efforts.
Quoted 5 lines
Quoted 2 lines
Is this something worth doing, and if yes, is this in the right direction?
Overall I think it is a good idea to slim down
sha1_file.c*if* we can keep the exposed surface area small enough.
What do you mean by "keep the exposed surface area small enough"? If you mean the total number of exposed functions in sha1_file and pack (once everything is done), I think it will be almost the same as that currently in sha1_file.
find_pack_entry() and has_packed_and_bad() (not yet in this patch set) might need to be changed from static to global, but those are the only 2 I can think of. Anyway, I'll report the functions that need to be changed from static to global at the end.
During this patch set, there might be some functions that need to be temporarily made global, but those are reverted to static in the end.
Quoted 3 lines
I wonder if the names "pack.[ch]" communicate well that these are "object access layer that is about reading from packfiles". The writer side is called "pack-objects.[ch]".
This file will end up being slightly broader than reading from packfiles
- in particular, things like pack_report() (reporting some statistics not only on the in-repo packfiles themselves) and parse_pack_index() (which parses an idx file obtained through http) are there too. Hence the generic name, but I agree that there might be a better name (or better set of names).
Quoted 7 lines
This may have to make some symbols that used to be private to the "object access" layer, which was what
sha1_file.cwas about, global symbols. After moving things around, do we end up exposing too many implementation details to the world outside the "object access" layer? I'd assume they are limited to the resultingpack.hand it would be OK as long as nobody other thansha1_file.candpack.cwould inculde it, though.
As stated above, I don't think so, but I'll make a list of the functions needing to be made global.
Quoted 7 lines · Jonathan Tan
What do you mean by "keep the exposed surface area small enough"? If you mean the total number of exposed functions in
sha1_fileand pack (once everything is done), I think it will be almost the same as that currently insha1_file.
...
During this patch set, there might be some functions that need to be temporarily made global, but those are reverted to static in the end.
That is exactly what I meant.
Quoted 2 lines
As stated above, I don't think so, but I'll make a list of the functions needing to be made global.
Good.
Thanks.
v2 cover Move exported packfile funcs to its own file
In reply to Jonathan TanHere is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like freshen_packed_object() that are used only by non-pack-specific functions are not moved.
In the end, 3 functions needed to be made global. They are find_pack_entry(), mark_bad_packed_object(), and has_packed_and_bad().
Of the 3, find_pack_entry() is probably legitimately promoted. But I think that the latter two functions needing to be accessed from sha1_file.c points to a design that could be improved - they are only used when packed_object_info() detects corruption, and used for marking as bad and printing messages to the user respectively, which packed_object_info() should probably do itself. But I have not made this change in this patch set.
(Other than the 3 functions above, there are some variables and functions that are temporarily made global, but reduced back to static when the wide scope is no longer needed.)
Jonathan Tan (25):
pack: move pack name-related functions
pack: move static state variables
pack: move pack_report()
pack: move open_pack_index(), parse_pack_index()
pack: move release_pack_memory()
pack: move pack-closing functions
pack: move use_pack()
pack: move unuse_pack()
pack: move add_packed_git()
pack: move install_packed_git()
pack: move {,re}prepare_packed_git and approximate_object_count
pack: move unpack_object_header()
pack: move get_size_from_delta()
pack: move unpack_object_header()
sha1_file: set whence in storage-specific info fn
sha1_file: remove read_packed_sha1()
pack: move packed_object_info(), unpack_entry()
pack: move nth_packed_object_{sha1,oid}
pack: move check_pack_index_ptr(), nth_packed_object_offset()
pack: move find_pack_entry_one(), is_pack_valid()
pack: move find_sha1_pack()
pack: move find_pack_entry() and make it global
pack: move has_sha1_pack()
pack: move has_pack_index()
pack: move for_each_packed_object()
Makefile | 1 + builtin/am.c | 1 + builtin/cat-file.c | 1 + builtin/clone.c | 1 + builtin/count-objects.c | 1 + builtin/fetch.c | 1 + builtin/gc.c | 1 + builtin/merge.c | 1 + builtin/pack-redundant.c | 1 + builtin/prune-packed.c | 1 + cache.h | 122 +-- connected.c | 1 + diff.c | 1 + git-compat-util.h | 2 - http-backend.c | 1 + http-push.c | 1 + http-walker.c | 1 + pack.h | 137 +++ packfile.c | 1905 +++++++++++++++++++++++++++++++++++ path.c | 1 + reachable.c | 1 + revision.c | 1 + server-info.c | 1 + sha1_file.c | 2484 ++++++---------------------------------------- sha1_name.c | 1 + streaming.c | 1 + 26 files changed, 2350 insertions(+), 2321 deletions(-) create mode 100644 packfile.c
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 1/25 pack: move pack name-related functions
Currently, sha1_file.c and cache.h contain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.
Create a new file, packfile.c, to hold all packfile-related functions currently in sha1_file.c, and designate pack.h to hold these packfile-related functions.
In this commit, the pack name-related functions are moved. Subsequent commits will move the other functions.
--- Makefile | 1 + builtin/pack-redundant.c | 1 + cache.h | 23 ----------------------- pack.h | 23 +++++++++++++++++++++++ packfile.c | 23 +++++++++++++++++++++++ sha1_file.c | 22 ---------------------- 6 files changed, 48 insertions(+), 45 deletions(-) create mode 100644 packfile.c
Makefile +1 -0
@@ -816,6 +816,7 @@ LIB_OBJS += notes-merge.o LIB_OBJS += notes-utils.o LIB_OBJS += object.o LIB_OBJS += oidset.o +LIB_OBJS += packfile.o LIB_OBJS += pack-bitmap.o LIB_OBJS += pack-bitmap-write.o LIB_OBJS += pack-check.o
builtin/pack-redundant.c +1 -0
@@ -7,6 +7,7 @@*/ #include "builtin.h" +#include "pack.h" #define BLKSIZE 512
cache.h +0 -23
@@ -902,20 +902,6 @@ extern void check_repository_format(void);*/ extern const char *sha1_file_name (const unsigned char *sha1); -/* -* Return the name of the (local) packfile with the specified sha1 in -* its name. The return value is a pointer to memory that is -* overwritten each time this function is called. -*/ -extern char *sha1_pack_name (const unsigned char *sha1); - -/* -* Return the name of the (local) pack index file with the specified -* sha1 in its name. The return value is a pointer to memory that is -* overwritten each time this function is called. -*/ -extern char *sha1_pack_index_name (const unsigned char *sha1); -/* * Return an abbreviated sha1 unique within this repository's object database. * The result will be at least `len` characters long, and will be NUL
@@ -1648,15 +1634,6 @@ extern void pack_report(void);*/ extern int odb_mkstemp (struct strbuf *template,const char *pattern); -/* -* Generate the filename to be used for a pack file with checksum "sha1" and -* extension "ext". The result is written into the strbuf "buf", overwriting -* any existing contents. A pointer to buf->buf is returned as a convenience. -* -* Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" -*/ -extern char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); -/* * Create a pack .keep file named "name" (which should generally be the output * of odb_pack_name). Returns a file descriptor opened for writing, or -1 on
pack.h +23 -0
@@ -101,4 +101,27 @@ extern int read_pack_header(int fd, struct pack_header *);extern struct sha1file *create_tmp_packfile (char **pack_tmp_name);extern void finish_tmp_packfile (struct strbuf *name_buffer,const char *pack_tmp_name,struct pack_idx_entry **written_list,uint32_t nr_written,struct pack_idx_option *pack_idx_opts,unsigned char sha1[]); +/* +* Generate the filename to be used for a pack file with checksum "sha1" and +* extension "ext". The result is written into the strbuf "buf", overwriting +* any existing contents. A pointer to buf->buf is returned as a convenience. +* +* Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" +*/ +extern char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); + +/* +* Return the name of the (local) packfile with the specified sha1 in +* its name. The return value is a pointer to memory that is +* overwritten each time this function is called. +*/ +extern char *sha1_pack_name (const unsigned char *sha1); + +/* +* Return the name of the (local) pack index file with the specified +* sha1 in its name. The return value is a pointer to memory that is +* overwritten each time this function is called. +*/ +extern char *sha1_pack_index_name (const unsigned char *sha1); +#endif
packfile.c new file +23 -0
@@ -0,0 +1,23 @@ +#include "cache.h" + +char *odb_pack_name (struct strbuf *buf, +const unsigned char *sha1, +const char *ext) +{ +strbuf_reset (buf); +strbuf_addf (buf,"%s/pack/pack-%s.%s" ,get_object_directory (), +sha1_to_hex (sha1), ext); +return buf->buf; +} + +char *sha1_pack_name (const unsigned char *sha1) +{ +static struct strbuf buf =STRBUF_INIT ; +return odb_pack_name (&buf, sha1,"pack" ); +} + +char *sha1_pack_index_name (const unsigned char *sha1) +{ +static struct strbuf buf =STRBUF_INIT ; +return odb_pack_name (&buf, sha1,"idx" ); +}
sha1_file.c +0 -22
@@ -278,28 +278,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,return buf->buf; } -char *odb_pack_name (struct strbuf *buf, -const unsigned char *sha1, -const char *ext) -{ -strbuf_reset (buf); -strbuf_addf (buf,"%s/pack/pack-%s.%s" ,get_object_directory (), -sha1_to_hex (sha1), ext); -return buf->buf; -} - -char *sha1_pack_name (const unsigned char *sha1) -{ -static struct strbuf buf =STRBUF_INIT ; -return odb_pack_name (&buf, sha1,"pack" ); -} - -char *sha1_pack_index_name (const unsigned char *sha1) -{ -static struct strbuf buf =STRBUF_INIT ; -return odb_pack_name (&buf, sha1,"idx" ); -} -struct alternate_object_database *alt_odb_list;static struct alternate_object_database **alt_odb_tail;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 2/25 pack: move static state variables
In reply to Jonathan Tansha1_file.c declares some static variables that store packfile-related state. Move them to packfile.c.
They are temporarily made global, but subsequent commits will restore their scope back to static.
--- pack.h | 9 +++++++++ packfile.c | 14 ++++++++++++++ sha1_file.c | 13 ------------- 3 files changed, 23 insertions(+), 13 deletions(-)
pack.h +9 -0
@@ -124,4 +124,13 @@ extern char *sha1_pack_name(const unsigned char *sha1);*/ extern char *sha1_pack_index_name (const unsigned char *sha1); +extern unsigned int pack_used_ctr; +extern unsigned int pack_mmap_calls; +extern unsigned int peak_pack_open_windows; +extern unsigned int pack_open_windows; +extern unsigned int pack_open_fds; +extern unsigned int pack_max_fds; +extern size_t peak_pack_mapped; +extern size_t pack_mapped; +#endif
packfile.c +14 -0
@@ -1,4 +1,5 @@#include "cache.h" +#include "mru.h" char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1,
@@ -21,3 +22,16 @@ char *sha1_pack_index_name(const unsigned char *sha1)static struct strbuf buf =STRBUF_INIT ;return odb_pack_name (&buf, sha1,"idx" ); } + +unsigned int pack_used_ctr; +unsigned int pack_mmap_calls; +unsigned int peak_pack_open_windows; +unsigned int pack_open_windows; +unsigned int pack_open_fds; +unsigned int pack_max_fds; +size_t peak_pack_mapped; +size_t pack_mapped; +struct packed_git *packed_git; + +static struct mru packed_git_mru_storage; +struct mru *packed_git_mru = &packed_git_mru_storage;
sha1_file.c +0 -13
@@ -682,19 +682,6 @@ static int has_loose_object(const unsigned char *sha1)return check_and_freshen (sha1,0 ); } -static unsigned int pack_used_ctr; -static unsigned int pack_mmap_calls; -static unsigned int peak_pack_open_windows; -static unsigned int pack_open_windows; -static unsigned int pack_open_fds; -static unsigned int pack_max_fds; -static size_t peak_pack_mapped; -static size_t pack_mapped; -struct packed_git *packed_git; - -static struct mru packed_git_mru_storage; -struct mru *packed_git_mru = &packed_git_mru_storage; -void pack_report (void ) { fprintf(stderr,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 3/25 pack: move pack_report()
In reply to Jonathan Tan--- cache.h | 2 -- pack.h | 2 ++ packfile.c | 24 ++++++++++++++++++++++++ sha1_file.c | 24 ------------------------ 4 files changed, 26 insertions(+), 26 deletions(-)
cache.h +0 -2
@@ -1624,8 +1624,6 @@ unsigned long approximate_object_count(void);extern struct packed_git *find_sha1_pack (const unsigned char *sha1,struct packed_git *packs); -extern void pack_report (void ); -/* * Create a temporary file rooted in the object database directory, or * die on failure. The filename is taken from "pattern", which should have the
pack.h +2 -0
@@ -133,4 +133,6 @@ extern unsigned int pack_max_fds;extern size_t peak_pack_mapped;extern size_t pack_mapped; +extern void pack_report (void ); +#endif
packfile.c +24 -0
@@ -35,3 +35,27 @@ struct packed_git *packed_git;static struct mru packed_git_mru_storage;struct mru *packed_git_mru = &packed_git_mru_storage; + +#define SZ_FMT PRIuMAX +static inline uintmax_t sz_fmt (size_t s) {return s; } + +void pack_report (void ) +{ +fprintf (stderr, +"pack_report: getpagesize() = %10" SZ_FMT "\n" +"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" +"pack_report: core.packedGitLimit = %10" SZ_FMT "\n" , +sz_fmt (getpagesize ()), +sz_fmt (packed_git_window_size), +sz_fmt (packed_git_limit)); +fprintf (stderr, +"pack_report: pack_used_ctr = %10u\n" +"pack_report: pack_mmap_calls = %10u\n" +"pack_report: pack_open_windows = %10u / %10u\n" +"pack_report: pack_mapped = " +"%10" SZ_FMT " / %10" SZ_FMT "\n" , + pack_used_ctr, + pack_mmap_calls, + pack_open_windows, peak_pack_open_windows, +sz_fmt (pack_mapped),sz_fmt (peak_pack_mapped)); +}
sha1_file.c +0 -24
@@ -29,9 +29,6 @@#include "mergesort.h" #include "quote.h" -#define SZ_FMT PRIuMAX -static inline uintmax_t sz_fmt (size_t s) {return s; } -const unsigned char null_sha1[20 ];const struct object_id null_oid;const struct object_id empty_tree_oid = {
@@ -682,27 +679,6 @@ static int has_loose_object(const unsigned char *sha1)return check_and_freshen (sha1,0 ); } -void pack_report (void ) -{ -fprintf (stderr, -"pack_report: getpagesize() = %10" SZ_FMT "\n" -"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" -"pack_report: core.packedGitLimit = %10" SZ_FMT "\n" , -sz_fmt (getpagesize ()), -sz_fmt (packed_git_window_size), -sz_fmt (packed_git_limit)); -fprintf (stderr, -"pack_report: pack_used_ctr = %10u\n" -"pack_report: pack_mmap_calls = %10u\n" -"pack_report: pack_open_windows = %10u / %10u\n" -"pack_report: pack_mapped = " -"%10" SZ_FMT " / %10" SZ_FMT "\n" , - pack_used_ctr, - pack_mmap_calls, - pack_open_windows, peak_pack_open_windows, -sz_fmt (pack_mapped),sz_fmt (peak_pack_mapped)); -} -/* * Open and mmap the index file at path, perform a couple of * consistency checks, then record its information to p. Return 0 on
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 4/25 pack: move open_pack_index(), parse_pack_index()
In reply to Jonathan Tanalloc_packed_git() in packfile.c is duplicated from sha1_file.c. In a subsequent commit, alloc_packed_git() will be removed from sha1_file.c.
--- builtin/count-objects.c | 1 + cache.h | 8 --- pack.h | 8 +++ packfile.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++ sha1_file.c | 140 --------------------------------------------- sha1_name.c | 1 + 6 files changed, 159 insertions(+), 148 deletions(-)
builtin/count-objects.c +1 -0
@@ -10,6 +10,7 @@#include "builtin.h" #include "parse-options.h" #include "quote.h" +#include "pack.h" static unsigned long garbage;static off_t size_garbage;
cache.h +0 -8
@@ -1603,8 +1603,6 @@ struct pack_entry {struct packed_git *p; }; -extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); -/* A hook to report invalid files in pack directory */ #define PACKDIR_FILE_PACK 1#define PACKDIR_FILE_IDX 2
@@ -1639,12 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* mmap the index file for the specified packfile (if it is not -* already mmapped). Return 0 on success. -*/ -extern int open_pack_index (struct packed_git *); -/* * munmap the index file for the specified packfile (if it is * currently mmapped).
pack.h +8 -0
@@ -135,4 +135,12 @@ extern size_t pack_mapped;extern void pack_report (void ); +/* +* mmap the index file for the specified packfile (if it is not +* already mmapped). Return 0 on success. +*/ +extern int open_pack_index (struct packed_git *); + +extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +#endif
packfile.c +149 -0
157 lines not on this page · raw message
sha1_file.c +0 -140
152 lines not on this page · raw message
sha1_name.c +1 -0
@@ -9,6 +9,7 @@#include "remote.h" #include "dir.h" #include "sha1-array.h" +#include "pack.h" static int get_sha1_oneline (const char *,unsigned char *,struct commit_list *);
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 5/25 pack: move release_pack_memory()
In reply to Jonathan TanThe function unuse_one_window() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.
--- git-compat-util.h | 2 -- pack.h | 4 ++++ packfile.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ sha1_file.c | 49 ------------------------------------------------- 4 files changed, 53 insertions(+), 51 deletions(-)
git-compat-util.h +0 -2
@@ -749,8 +749,6 @@ const char *inet_ntop(int af, const void *src, char *dst, size_t size);extern int git_atexit (void (*handler)(void ));#endif -extern void release_pack_memory (size_t ); -typedef void (*try_to_free_t )(size_t );extern try_to_free_t set_try_to_free_routine (try_to_free_t );
pack.h +4 -0
@@ -143,4 +143,8 @@ extern int open_pack_index(struct packed_git *);extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +extern int unuse_one_window (struct packed_git *current); + +extern void release_pack_memory (size_t ); +#endif
packfile.c +49 -0
52 lines not on this page · raw message
sha1_file.c +0 -49
55 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 6/25 pack: move pack-closing functions
In reply to Jonathan TanThe function close_pack_fd() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.
--- builtin/am.c | 1 + builtin/clone.c | 1 + builtin/fetch.c | 1 + builtin/merge.c | 1 + cache.h | 8 -------- pack.h | 9 +++++++++ packfile.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ sha1_file.c | 55 ------------------------------------------------------- 8 files changed, 67 insertions(+), 63 deletions(-)
builtin/am.c +1 -0
@@ -31,6 +31,7 @@#include "mailinfo.h" #include "apply.h" #include "string-list.h" +#include "pack.h" /** * Returns 1 if the file is empty or does not exist, 0 otherwise.
builtin/clone.c +1 -0
@@ -25,6 +25,7 @@#include "remote.h" #include "run-command.h" #include "connected.h" +#include "pack.h" /* * Overall FIXMEs:
builtin/fetch.c +1 -0
@@ -17,6 +17,7 @@#include "connected.h" #include "argv-array.h" #include "utf8.h" +#include "pack.h" static const char *const builtin_fetch_usage[] = {N_ ("git fetch [<options>] [<repository> [<refspec>...]]" ),
builtin/merge.c +1 -0
@@ -32,6 +32,7 @@#include "gpg-interface.h" #include "sequencer.h" #include "string-list.h" +#include "pack.h" #define DEFAULT_TWOHEAD (1<<0)#define DEFAULT_OCTOPUS (1<<1)
cache.h +0 -8
@@ -1637,15 +1637,7 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* munmap the index file for the specified packfile (if it is -* currently mmapped). -*/ -extern void close_pack_index (struct packed_git *); -extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *); -extern void close_pack_windows (struct packed_git *); -extern void close_all_packs (void );extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
pack.h +9 -0
@@ -147,4 +147,13 @@ extern int unuse_one_window(struct packed_git *current);extern void release_pack_memory (size_t ); +extern void close_pack_windows (struct packed_git *); +extern int close_pack_fd (struct packed_git *); +/* +* munmap the index file for the specified packfile (if it is +* currently mmapped). +*/ +extern void close_pack_index (struct packed_git *); +extern void close_all_packs (void ); +#endif
packfile.c +54 -0
57 lines not on this page · raw message
sha1_file.c +0 -55
67 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 7/25 pack: move use_pack()
In reply to Jonathan TanThe function open_packed_git() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.
--- cache.h | 1 - pack.h | 14 +-- packfile.c | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- sha1_file.c | 285 -------------------------------------------------------- streaming.c | 1 + 5 files changed, 299 insertions(+), 305 deletions(-)
cache.h +0 -1
@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
pack.h +4 -10
@@ -124,14 +124,7 @@ extern char *sha1_pack_name(const unsigned char *sha1);*/ extern char *sha1_pack_index_name (const unsigned char *sha1); -extern unsigned int pack_used_ctr; -extern unsigned int pack_mmap_calls; -extern unsigned int peak_pack_open_windows; -extern unsigned int pack_open_windows;extern unsigned int pack_open_fds; -extern unsigned int pack_max_fds; -extern size_t peak_pack_mapped; -extern size_t pack_mapped;extern void pack_report (void );
@@ -143,12 +136,9 @@ extern int open_pack_index(struct packed_git *);extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); -extern int unuse_one_window (struct packed_git *current); -extern void release_pack_memory (size_t );extern void close_pack_windows (struct packed_git *); -extern int close_pack_fd (struct packed_git *);/* * munmap the index file for the specified packfile (if it is * currently mmapped).
@@ -156,4 +146,8 @@ extern int close_pack_fd(struct packed_git *);extern void close_pack_index (struct packed_git *);extern void close_all_packs (void ); +extern int open_packed_git (struct packed_git *p); + +extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *); +#endif
packfile.c +294 -9
325 lines not on this page · raw message
sha1_file.c +0 -285
297 lines not on this page · raw message
streaming.c +1 -0
@@ -3,6 +3,7 @@*/ #include "cache.h" #include "streaming.h" +#include "pack.h" enum input_source { stream_error =-1 ,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 8/25 pack: move unuse_pack()
In reply to Jonathan Tan--- cache.h | 1 - pack.h | 1 + packfile.c | 9 +++++++++ sha1_file.c | 9 --------- 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -1
@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
pack.h +1 -0
@@ -149,5 +149,6 @@ extern void close_all_packs(void);extern int open_packed_git (struct packed_git *p);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *); +extern void unuse_pack (struct pack_window **);#endif
packfile.c +9 -0
@@ -596,3 +596,12 @@ unsigned char *use_pack(struct packed_git *p, *left = win->len -xsize_t (offset);return win->base + offset; } + +void unuse_pack (struct pack_window **w_cursor) +{ +struct pack_window *w = *w_cursor; +if (w) { + w->inuse_cnt--; + *w_cursor =NULL ; + } +}
sha1_file.c +0 -9
@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,return ret; } -void unuse_pack (struct pack_window **w_cursor) -{ -struct pack_window *w = *w_cursor; -if (w) { - w->inuse_cnt--; - *w_cursor =NULL ; - } -} -static struct packed_git *alloc_packed_git (int extra) {struct packed_git *p =xmalloc (st_add (sizeof (*p), extra));
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 9/25 pack: move add_packed_git()
In reply to Jonathan Tan--- cache.h | 1 - connected.c | 1 + pack.h | 1 + packfile.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ sha1_file.c | 61 ------------------------------------------------------------- 5 files changed, 55 insertions(+), 62 deletions(-)
cache.h +0 -1
@@ -1638,7 +1638,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);extern int odb_pack_keep (const char *name);extern void clear_delta_base_cache (void ); -extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);/* * Make sure that a pointer access into an mmap'd index file is within bounds,
connected.c +1 -0
@@ -3,6 +3,7 @@#include "sigchain.h" #include "connected.h" #include "transport.h" +#include "pack.h" /* * If we feed all the commits we want to verify to this command
pack.h +1 -0
@@ -150,5 +150,6 @@ extern int open_packed_git(struct packed_git *p);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void unuse_pack (struct pack_window **); +extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);#endif
packfile.c +53 -0
56 lines not on this page · raw message
sha1_file.c +0 -61
67 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 10/25 pack: move install_packed_git()
In reply to Jonathan Tan--- cache.h | 1 - pack.h | 4 ++-- packfile.c | 11 ++++++++++- sha1_file.c | 9 --------- 4 files changed, 12 insertions(+), 13 deletions(-)
cache.h +0 -1
@@ -1611,7 +1611,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);extern void prepare_packed_git (void );extern void reprepare_packed_git (void ); -extern void install_packed_git (struct packed_git *pack);/* * Give a rough count of objects in the repository. This sacrifices accuracy
pack.h +2 -2
@@ -124,8 +124,6 @@ extern char *sha1_pack_name(const unsigned char *sha1);*/ extern char *sha1_pack_index_name (const unsigned char *sha1); -extern unsigned int pack_open_fds; -extern void pack_report (void );/*
@@ -152,4 +150,6 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_textern void unuse_pack (struct pack_window **);extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +extern void install_packed_git (struct packed_git *pack); +#endif
packfile.c +10 -1
@@ -28,7 +28,7 @@ static unsigned int pack_used_ctr;static unsigned int pack_mmap_calls;static unsigned int peak_pack_open_windows;static unsigned int pack_open_windows; -unsigned int pack_open_fds; +static unsigned int pack_open_fds;static unsigned int pack_max_fds;static size_t peak_pack_mapped;static size_t pack_mapped;
@@ -658,3 +658,12 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)hashclr (p->sha1);return p; } + +void install_packed_git (struct packed_git *pack) +{ +if (pack->pack_fd !=-1 ) + pack_open_fds++; + + pack->next = packed_git; + packed_git = pack; +}
sha1_file.c +0 -9
@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,return ret; } -void install_packed_git (struct packed_git *pack) -{ -if (pack->pack_fd !=-1 ) - pack_open_fds++; - - pack->next = packed_git; - packed_git = pack; -} -void (*report_garbage)(unsigned seen_bits,const char *path); static void report_helper(const struct string_list *list,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 11/25 pack: move {,re}prepare_packed_git and approximate_object_count
In reply to Jonathan Tan--- builtin/gc.c | 1 + cache.h | 15 ---- http-backend.c | 1 + pack.h | 15 ++++ packfile.c | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ path.c | 1 + server-info.c | 1 + sha1_file.c | 214 -------------------------------------------------------- 8 files changed, 235 insertions(+), 229 deletions(-)
builtin/gc.c +1 -0
@@ -19,6 +19,7 @@#include "sigchain.h" #include "argv-array.h" #include "commit.h" +#include "pack.h" #define FAILED_RUN "failed to run %s"
cache.h +0 -15
@@ -1603,21 +1603,6 @@ struct pack_entry {struct packed_git *p; }; -/* A hook to report invalid files in pack directory */ -#define PACKDIR_FILE_PACK 1 -#define PACKDIR_FILE_IDX 2 -#define PACKDIR_FILE_GARBAGE 4 -extern void (*report_garbage)(unsigned seen_bits,const char *path); - -extern void prepare_packed_git (void ); -extern void reprepare_packed_git (void ); - -/* -* Give a rough count of objects in the repository. This sacrifices accuracy -* for speed. -*/ -unsigned long approximate_object_count (void ); -extern struct packed_git *find_sha1_pack (const unsigned char *sha1,struct packed_git *packs);
http-backend.c +1 -0
@@ -9,6 +9,7 @@#include "string-list.h" #include "url.h" #include "argv-array.h" +#include "pack.h" static const char content_type[] ="Content-Type" ;static const char content_length[] ="Content-Length" ;
pack.h +15 -0
@@ -152,4 +152,19 @@ extern struct packed_git *add_packed_git(const char *path, size_t path_len, intextern void install_packed_git (struct packed_git *pack); +/* A hook to report invalid files in pack directory */ +#define PACKDIR_FILE_PACK 1 +#define PACKDIR_FILE_IDX 2 +#define PACKDIR_FILE_GARBAGE 4 +extern void (*report_garbage)(unsigned seen_bits,const char *path); + +extern void prepare_packed_git (void ); +extern void reprepare_packed_git (void ); + +/* +* Give a rough count of objects in the repository. This sacrifices accuracy +* for speed. +*/ +unsigned long approximate_object_count (void ); +#endif
packfile.c +216 -0
225 lines not on this page · raw message
path.c +1 -0
@@ -9,6 +9,7 @@#include "worktree.h" #include "submodule-config.h" #include "path.h" +#include "pack.h" static int get_st_mode_bits (const char *path,int *mode) {
server-info.c +1 -0
@@ -3,6 +3,7 @@#include "object.h" #include "commit.h" #include "tag.h" +#include "pack.h" /* * Create the file "path" by writing to a temporary file and renaming
sha1_file.c +0 -214
220 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 12/25 pack: move unpack_object_header()
In reply to Jonathan Tan--- cache.h | 1 - pack.h | 2 ++ packfile.c | 25 +++++++++++++++++++++++++ sha1_file.c | 25 ------------------------- 4 files changed, 27 insertions(+), 26 deletions(-)
cache.h +0 -1
@@ -1661,7 +1661,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)extern int is_pack_valid (struct packed_git *);extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *); -extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep);extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *);
pack.h +2 -0
@@ -167,4 +167,6 @@ extern void reprepare_packed_git(void);*/ unsigned long approximate_object_count (void ); +extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep); +#endif
packfile.c +25 -0
@@ -883,3 +883,28 @@ void reprepare_packed_git(void) prepare_packed_git_run_once =0 ;prepare_packed_git (); } + +unsigned long unpack_object_header_buffer (const unsigned char *buf, +unsigned long len,enum object_type *type,unsigned long *sizep) +{ +unsigned shift; +unsigned long size, c; +unsigned long used =0 ; + + c = buf[used++]; + *type = (c >>4 ) &7 ; + size = c &15 ; + shift =4 ; +while (c &0x80 ) { +if (len <= used ||bitsizeof (long) <= shift) { +error ("bad object header" ); + size = used =0 ; +break ; + } + c = buf[used++]; + size += (c &0x7f ) << shift; + shift +=7 ; + } + *sizep = size; +return used; +}
sha1_file.c +0 -25
@@ -913,31 +913,6 @@ void *map_sha1_file(const unsigned char *sha1, unsigned long *size)return map_sha1_file_1 (NULL , sha1, size); } -unsigned long unpack_object_header_buffer (const unsigned char *buf, -unsigned long len,enum object_type *type,unsigned long *sizep) -{ -unsigned shift; -unsigned long size, c; -unsigned long used =0 ; - - c = buf[used++]; - *type = (c >>4 ) &7 ; - size = c &15 ; - shift =4 ; -while (c &0x80 ) { -if (len <= used ||bitsizeof (long) <= shift) { -error ("bad object header" ); - size = used =0 ; -break ; - } - c = buf[used++]; - size += (c &0x7f ) << shift; - shift +=7 ; - } - *sizep = size; -return used; -} -static int unpack_sha1_short_header (git_zstream *stream,unsigned char *map,unsigned long mapsize,void *buffer,unsigned long bufsiz)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 13/25 pack: move get_size_from_delta()
In reply to Jonathan Tan--- cache.h | 1 - pack.h | 1 + packfile.c | 40 ++++++++++++++++++++++++++++++++++++++++ sha1_file.c | 39 --------------------------------------- 4 files changed, 41 insertions(+), 40 deletions(-)
cache.h +0 -1
@@ -1661,7 +1661,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)extern int is_pack_valid (struct packed_git *);extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *); -extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *);/*
pack.h +1 -0
@@ -168,5 +168,6 @@ extern void reprepare_packed_git(void);unsigned long approximate_object_count (void );extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep); +extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );#endif
packfile.c +40 -0
@@ -3,6 +3,7 @@#include "pack.h" #include "dir.h" #include "mergesort.h" +#include "delta.h" char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1,
@@ -908,3 +909,42 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf, *sizep = size;return used; } + +unsigned long get_size_from_delta (struct packed_git *p, +struct pack_window **w_curs, +off_t curpos) +{ +const unsigned char *data; +unsigned char delta_head[20 ], *in; +git_zstream stream; +int st; + +memset (&stream,0 ,sizeof (stream)); + stream.next_out = delta_head; + stream.avail_out =sizeof (delta_head); + +git_inflate_init (&stream); +do { + in =use_pack (p, w_curs, curpos, &stream.avail_in); + stream.next_in = in; + st =git_inflate (&stream,Z_FINISH ); + curpos += stream.next_in - in; + }while ((st ==Z_OK || st ==Z_BUF_ERROR ) && + stream.total_out <sizeof (delta_head)); +git_inflate_end (&stream); +if ((st !=Z_STREAM_END ) && stream.total_out !=sizeof (delta_head)) { +error ("delta data unpack-initial failed" ); +return 0 ; + } + +/* Examine the initial part of the delta to figure out +* the result size. +*/ + data = delta_head; + +/* ignore base size */ +get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); + +/* Read the result size */ +return get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); +}
sha1_file.c +0 -39
@@ -1099,45 +1099,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -unsigned long get_size_from_delta (struct packed_git *p, -struct pack_window **w_curs, -off_t curpos) -{ -const unsigned char *data; -unsigned char delta_head[20 ], *in; -git_zstream stream; -int st; - -memset (&stream,0 ,sizeof (stream)); - stream.next_out = delta_head; - stream.avail_out =sizeof (delta_head); - -git_inflate_init (&stream); -do { - in =use_pack (p, w_curs, curpos, &stream.avail_in); - stream.next_in = in; - st =git_inflate (&stream,Z_FINISH ); - curpos += stream.next_in - in; - }while ((st ==Z_OK || st ==Z_BUF_ERROR ) && - stream.total_out <sizeof (delta_head)); -git_inflate_end (&stream); -if ((st !=Z_STREAM_END ) && stream.total_out !=sizeof (delta_head)) { -error ("delta data unpack-initial failed" ); -return 0 ; - } - -/* Examine the initial part of the delta to figure out -* the result size. -*/ - data = delta_head; - -/* ignore base size */ -get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); - -/* Read the result size */ -return get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); -} - static off_t get_delta_base(struct packed_git *p, struct pack_window **w_curs, off_t *curpos,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 14/25 pack: move unpack_object_header()
In reply to Jonathan Tan--- cache.h | 1 - pack.h | 1 + packfile.c | 26 ++++++++++++++++++++++++++ sha1_file.c | 26 -------------------------- 4 files changed, 27 insertions(+), 27 deletions(-)
cache.h +0 -1
@@ -1661,7 +1661,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)extern int is_pack_valid (struct packed_git *);extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *); -extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *);/* * Iterate over the files in the loose-object parts of the object
pack.h +1 -0
@@ -169,5 +169,6 @@ unsigned long approximate_object_count(void);extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep);extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t ); +extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *);#endif
packfile.c +26 -0
@@ -948,3 +948,29 @@ unsigned long get_size_from_delta(struct packed_git *p,/* Read the result size */ return get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); } + +int unpack_object_header (struct packed_git *p, +struct pack_window **w_curs, +off_t *curpos, +unsigned long *sizep) +{ +unsigned char *base; +unsigned long left; +unsigned long used; +enum object_type type; + +/* use_pack() assures us we have [base, base + 20) available +* as a range that we can look at. (Its actually the hash +* size that is assured.) With our object header encoding +* the maximum deflated object size is 2^137, which is just +* insane, so we know won't exceed what we have been given. +*/ + base =use_pack (p, w_curs, *curpos, &left); + used =unpack_object_header_buffer (base, left, &type, sizep); +if (!used) { + type =OBJ_BAD ; + }else + *curpos += used; + +return type; +}
sha1_file.c +0 -26
@@ -1170,32 +1170,6 @@ static const unsigned char *get_delta_base_sha1(struct packed_git *p,return NULL ; } -int unpack_object_header (struct packed_git *p, -struct pack_window **w_curs, -off_t *curpos, -unsigned long *sizep) -{ -unsigned char *base; -unsigned long left; -unsigned long used; -enum object_type type; - -/* use_pack() assures us we have [base, base + 20) available -* as a range that we can look at. (Its actually the hash -* size that is assured.) With our object header encoding -* the maximum deflated object size is 2^137, which is just -* insane, so we know won't exceed what we have been given. -*/ - base =use_pack (p, w_curs, *curpos, &left); - used =unpack_object_header_buffer (base, left, &type, sizep); -if (!used) { - type =OBJ_BAD ; - }else - *curpos += used; - -return type; -} -static int retry_bad_packed_offset (struct packed_git *p,off_t obj_offset) {int type;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 15/25 sha1_file: set whence in storage-specific info fn
In reply to Jonathan TanMove the setting of oi->whence to sha1_loose_object_info() and packed_object_info(). This allows sha1_object_info_extended() to not need to know about the delta base cache.
--- sha1_file.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
sha1_file.c +6 -7
@@ -1473,6 +1473,9 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,hashclr (oi->delta_base_sha1); } + oi->whence =in_delta_base_cache (p, obj_offset) ?OI_DBCACHED : +OI_PACKED ; + out:unuse_pack (&w_curs);return type;
@@ -2002,6 +2005,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,if (oi->sizep == &size_scratch) oi->sizep =NULL ;strbuf_release (&hdrbuf); + oi->whence =OI_LOOSE ;return (status <0 ) ? status :0 ; }
@@ -2039,10 +2043,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,if (!find_pack_entry (real, &e)) {/* Most likely it's a loose object. */ -if (!sha1_loose_object_info (real, oi, flags)) { - oi->whence =OI_LOOSE ; +if (!sha1_loose_object_info (real, oi, flags))return 0 ; - }/* Not a loose object; someone else may have just packed it. */ if (flags &OBJECT_INFO_QUICK ) {
@@ -2065,10 +2067,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,if (rtype <0 ) {mark_bad_packed_object (e.p, real);return sha1_object_info_extended (real, oi,0 ); - }else if (in_delta_base_cache (e.p, e.offset)) { - oi->whence =OI_DBCACHED ; - }else { - oi->whence =OI_PACKED ; + }else if (oi->whence ==OI_PACKED ) { oi->u.packed.offset = e.offset; oi->u.packed.pack = e.p; oi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 16/25 sha1_file: remove read_packed_sha1()
In reply to Jonathan TanUse read_object() in its place instead. This avoids duplication of code.
This makes force_object_loose() slightly slower (because of a redundant check of loose object storage), but only in the error case.
--- sha1_file.c | 26 +------------------------- 1 file changed, 1 insertion(+), 25 deletions(-)
sha1_file.c +1 -25
@@ -2091,30 +2091,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)return type; } -static void *read_packed_sha1 (const unsigned char *sha1, -enum object_type *type,unsigned long *size) -{ -struct pack_entry e; -void *data; - -if (!find_pack_entry (sha1, &e)) -return NULL ; - data =cache_or_unpack_entry (e.p, e.offset, size, type); -if (!data) { -/* -* We're probably in deep shit, but let's try to fetch -* the required object anyway from another pack or loose. -* This should happen only in the presence of a corrupted -* pack, and is better than failing outright. -*/ -error ("failed to read object %s at offset %" PRIuMAX" from %s" , -sha1_to_hex (sha1), (uintmax_t )e.offset, e.p->pack_name); -mark_bad_packed_object (e.p, sha1); - data =read_object (sha1, type, size); - } -return data; -} -int pretend_sha1_file (void *buf,unsigned long len,enum object_type type,unsigned char *sha1) {
@@ -2497,7 +2473,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)if (has_loose_object (sha1))return 0 ; - buf =read_packed_sha1 (sha1, &type, &len); + buf =read_object (sha1, &type, &len);if (!buf)return error ("cannot read sha1_file for %s" ,sha1_to_hex (sha1)); hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %lu" ,typename (type), len) +1 ;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 17/25 pack: move packed_object_info(), unpack_entry()
In reply to Jonathan TanBoth sha1_file.c and packfile.c now need read_object(), so a copy of read_object() was created in packfile.c.
This patch makes both mark_bad_packed_object() and has_packed_and_bad() global. Unlike most of the other patches in this series, these 2 functions need to remain global.
--- cache.h | 7 - pack.h | 11 + packfile.c | 660 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ sha1_file.c | 676 ++---------------------------------------------------------- 4 files changed, 685 insertions(+), 669 deletions(-)
cache.h +0 -7
@@ -1186,9 +1186,6 @@ extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);extern int unpack_sha1_header (git_zstream *stream,unsigned char *map,unsigned long mapsize,void *buffer,unsigned long bufsiz);extern int parse_sha1_header (const char *hdr,unsigned long *sizep); -/* global flag to enable extra checks when accessing packed objects */ -extern int do_check_packed_object_crc; -extern int check_sha1_signature (const unsigned char *sha1,void *buf,unsigned long size,const char *type);extern int finalize_object_file (const char *tmpfile,const char *filename);
@@ -1621,8 +1618,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -extern void clear_delta_base_cache (void ); -/* * Make sure that a pointer access into an mmap'd index file is within bounds, * and can provide at least 8 bytes of data.
@@ -1660,7 +1655,6 @@ extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);extern off_t find_pack_entry_one (const unsigned char *sha1,struct packed_git *);extern int is_pack_valid (struct packed_git *); -extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *);/* * Iterate over the files in the loose-object parts of the object
@@ -1771,7 +1765,6 @@ struct object_info {/* Do not retry packed storage after checking packed and loose storage */ #define OBJECT_INFO_QUICK 8extern int sha1_object_info_extended (const unsigned char *,struct object_info *,unsigned flags); -extern int packed_object_info (struct packed_git *pack,off_t offset,struct object_info *);/* Dumb servers support */ extern int update_server_info (int );
pack.h +11 -0
@@ -171,4 +171,15 @@ extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigextern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *); +extern void clear_delta_base_cache (void ); + +/* global flag to enable extra checks when accessing packed objects */ +extern int do_check_packed_object_crc; + +extern int packed_object_info (struct packed_git *pack,off_t offset,struct object_info *); +extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *); + +extern void mark_bad_packed_object (struct packed_git *p,const unsigned char *sha1); +extern const struct packed_git *has_packed_and_bad (const unsigned char *sha1); +#endif
packfile.c +660 -0
669 lines not on this page · raw message
sha1_file.c +14 -662
700 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 18/25 pack: move nth_packed_object_{sha1,oid}
In reply to Jonathan Tan--- cache.h | 14 -------------- pack.h | 14 ++++++++++++++ packfile.c | 31 +++++++++++++++++++++++++++++++ sha1_file.c | 31 ------------------------------- 4 files changed, 45 insertions(+), 45 deletions(-)
cache.h +0 -14
@@ -1628,20 +1628,6 @@ extern int odb_pack_keep(const char *name);*/ extern void check_pack_index_ptr (const struct packed_git *p,const void *ptr); -/* -* Return the SHA-1 of the nth object within the specified packfile. -* Open the index if it is not already open. The return value points -* at the SHA-1 within the mmapped index. Return NULL if there is an -* error. -*/ -extern const unsigned char *nth_packed_object_sha1 (struct packed_git *,uint32_t n); -/* -* Like nth_packed_object_sha1, but write the data into the object specified by -* the the first argument. Returns the first argument on success, and NULL on -* error. -*/ -extern const struct object_id *nth_packed_object_oid (struct object_id *,struct packed_git *,uint32_t n); -/* * Return the offset of the nth object within the specified packfile. * The index must already be opened.
pack.h +14 -0
@@ -182,4 +182,18 @@ extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsignextern void mark_bad_packed_object (struct packed_git *p,const unsigned char *sha1);extern const struct packed_git *has_packed_and_bad (const unsigned char *sha1); +/* +* Return the SHA-1 of the nth object within the specified packfile. +* Open the index if it is not already open. The return value points +* at the SHA-1 within the mmapped index. Return NULL if there is an +* error. +*/ +extern const unsigned char *nth_packed_object_sha1 (struct packed_git *,uint32_t n); +/* +* Like nth_packed_object_sha1, but write the data into the object specified by +* the the first argument. Returns the first argument on success, and NULL on +* error. +*/ +extern const struct object_id *nth_packed_object_oid (struct object_id *,struct packed_git *,uint32_t n); +#endif
packfile.c +31 -0
@@ -1634,3 +1634,34 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,return data; } + +const unsigned char *nth_packed_object_sha1 (struct packed_git *p, +uint32_t n) +{ +const unsigned char *index = p->index_data; +if (!index) { +if (open_pack_index (p)) +return NULL ; + index = p->index_data; + } +if (n >= p->num_objects) +return NULL ; + index +=4 *256 ; +if (p->index_version ==1 ) { +return index +24 * n +4 ; + }else { + index +=8 ; +return index +20 * n; + } +} + +const struct object_id *nth_packed_object_oid (struct object_id *oid, +struct packed_git *p, +uint32_t n) +{ +const unsigned char *hash =nth_packed_object_sha1 (p, n); +if (!hash) +return NULL ; +hashcpy (oid->hash, hash); +return oid; +}
sha1_file.c +0 -31
@@ -1073,37 +1073,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -const unsigned char *nth_packed_object_sha1 (struct packed_git *p, -uint32_t n) -{ -const unsigned char *index = p->index_data; -if (!index) { -if (open_pack_index (p)) -return NULL ; - index = p->index_data; - } -if (n >= p->num_objects) -return NULL ; - index +=4 *256 ; -if (p->index_version ==1 ) { -return index +24 * n +4 ; - }else { - index +=8 ; -return index +20 * n; - } -} - -const struct object_id *nth_packed_object_oid (struct object_id *oid, -struct packed_git *p, -uint32_t n) -{ -const unsigned char *hash =nth_packed_object_sha1 (p, n); -if (!hash) -return NULL ; -hashcpy (oid->hash, hash); -return oid; -} -void check_pack_index_ptr (const struct packed_git *p,const void *vptr) {const unsigned char *ptr = vptr;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 19/25 pack: move check_pack_index_ptr(), nth_packed_object_offset()
In reply to Jonathan Tan--- cache.h | 16 ---------------- pack.h | 16 ++++++++++++++++ packfile.c | 33 +++++++++++++++++++++++++++++++++ sha1_file.c | 33 --------------------------------- 4 files changed, 49 insertions(+), 49 deletions(-)
cache.h +0 -16
@@ -1618,22 +1618,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* Make sure that a pointer access into an mmap'd index file is within bounds, -* and can provide at least 8 bytes of data. -* -* Note that this is only necessary for variable-length segments of the file -* (like the 64-bit extended offset table), as we compare the size to the -* fixed-length parts when we open the file. -*/ -extern void check_pack_index_ptr (const struct packed_git *p,const void *ptr); - -/* -* Return the offset of the nth object within the specified packfile. -* The index must already be opened. -*/ -extern off_t nth_packed_object_offset (const struct packed_git *,uint32_t n); -/* * If the object named sha1 is present in the specified packfile, * return its offset within the packfile; otherwise, return 0.
pack.h +16 -0
@@ -196,4 +196,20 @@ extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t*/ extern const struct object_id *nth_packed_object_oid (struct object_id *,struct packed_git *,uint32_t n); +/* +* Make sure that a pointer access into an mmap'd index file is within bounds, +* and can provide at least 8 bytes of data. +* +* Note that this is only necessary for variable-length segments of the file +* (like the 64-bit extended offset table), as we compare the size to the +* fixed-length parts when we open the file. +*/ +extern void check_pack_index_ptr (const struct packed_git *p,const void *ptr); + +/* +* Return the offset of the nth object within the specified packfile. +* The index must already be opened. +*/ +extern off_t nth_packed_object_offset (const struct packed_git *,uint32_t n); +#endif
packfile.c +33 -0
@@ -1665,3 +1665,36 @@ const struct object_id *nth_packed_object_oid(struct object_id *oid,hashcpy (oid->hash, hash);return oid; } + +void check_pack_index_ptr (const struct packed_git *p,const void *vptr) +{ +const unsigned char *ptr = vptr; +const unsigned char *start = p->index_data; +const unsigned char *end = start + p->index_size; +if (ptr < start) +die (_ ("offset before start of pack index for %s (corrupt index?)" ), + p->pack_name); +/* No need to check for underflow; .idx files must be at least 8 bytes */ +if (ptr >= end -8 ) +die (_ ("offset beyond end of pack index for %s (truncated index?)" ), + p->pack_name); +} + +off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) +{ +const unsigned char *index = p->index_data; + index +=4 *256 ; +if (p->index_version ==1 ) { +return ntohl (*((uint32_t *)(index +24 * n))); + }else { +uint32_t off; + index +=8 + p->num_objects * (20 +4 ); + off =ntohl (*((uint32_t *)(index +4 * n))); +if (!(off &0x80000000 )) +return off; + index += p->num_objects *4 + (off &0x7fffffff ) *8 ; +check_pack_index_ptr (p, index); +return (((uint64_t )ntohl (*((uint32_t *)(index +0 )))) <<32 ) | +ntohl (*((uint32_t *)(index +4 ))); + } +}
sha1_file.c +0 -33
@@ -1073,39 +1073,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -void check_pack_index_ptr (const struct packed_git *p,const void *vptr) -{ -const unsigned char *ptr = vptr; -const unsigned char *start = p->index_data; -const unsigned char *end = start + p->index_size; -if (ptr < start) -die (_ ("offset before start of pack index for %s (corrupt index?)" ), - p->pack_name); -/* No need to check for underflow; .idx files must be at least 8 bytes */ -if (ptr >= end -8 ) -die (_ ("offset beyond end of pack index for %s (truncated index?)" ), - p->pack_name); -} - -off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) -{ -const unsigned char *index = p->index_data; - index +=4 *256 ; -if (p->index_version ==1 ) { -return ntohl (*((uint32_t *)(index +24 * n))); - }else { -uint32_t off; - index +=8 + p->num_objects * (20 +4 ); - off =ntohl (*((uint32_t *)(index +4 * n))); -if (!(off &0x80000000 )) -return off; - index += p->num_objects *4 + (off &0x7fffffff ) *8 ; -check_pack_index_ptr (p, index); -return (((uint64_t )ntohl (*((uint32_t *)(index +0 )))) <<32 ) | -ntohl (*((uint32_t *)(index +4 ))); - } -} -off_t find_pack_entry_one (const unsigned char *sha1,struct packed_git *p) {
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 20/25 pack: move find_pack_entry_one(), is_pack_valid()
In reply to Jonathan Tan--- cache.h | 8 ------ pack.h | 10 ++++++-- packfile.c | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ sha1_file.c | 84 ------------------------------------------------------------ 4 files changed, 93 insertions(+), 94 deletions(-)
cache.h +0 -8
@@ -1618,14 +1618,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* If the object named sha1 is present in the specified packfile, -* return its offset within the packfile; otherwise, return 0. -*/ -extern off_t find_pack_entry_one (const unsigned char *sha1,struct packed_git *); - -extern int is_pack_valid (struct packed_git *); -/* * Iterate over the files in the loose-object parts of the object * directory "path", triggering the following callbacks:
pack.h +8 -2
@@ -144,8 +144,6 @@ extern void close_pack_windows(struct packed_git *);extern void close_pack_index (struct packed_git *);extern void close_all_packs (void ); -extern int open_packed_git (struct packed_git *p); -extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,unsigned long *);extern void unuse_pack (struct pack_window **);extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
@@ -212,4 +210,12 @@ extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);*/ extern off_t nth_packed_object_offset (const struct packed_git *,uint32_t n); +/* +* If the object named sha1 is present in the specified packfile, +* return its offset within the packfile; otherwise, return 0. +*/ +extern off_t find_pack_entry_one (const unsigned char *sha1,struct packed_git *); + +extern int is_pack_valid (struct packed_git *); +#endif
packfile.c +85 -0
94 lines not on this page · raw message
sha1_file.c +0 -84
90 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 21/25 pack: move find_sha1_pack()
In reply to Jonathan Tan--- cache.h | 3 --- http-push.c | 1 + http-walker.c | 1 + pack.h | 3 +++ packfile.c | 13 +++++++++++++ sha1_file.c | 13 ------------- 6 files changed, 18 insertions(+), 16 deletions(-)
cache.h +0 -3
@@ -1600,9 +1600,6 @@ struct pack_entry {struct packed_git *p; }; -extern struct packed_git *find_sha1_pack (const unsigned char *sha1, -struct packed_git *packs); -/* * Create a temporary file rooted in the object database directory, or * die on failure. The filename is taken from "pattern", which should have the
http-push.c +1 -0
@@ -11,6 +11,7 @@#include "list-objects.h" #include "sigchain.h" #include "argv-array.h" +#include "pack.h" #ifdef EXPAT_NEEDS_XMLPARSE_H #include <xmlparse.h>
http-walker.c +1 -0
@@ -4,6 +4,7 @@#include "http.h" #include "list.h" #include "transport.h" +#include "pack.h" struct alt_base {char *base;
pack.h +3 -0
@@ -218,4 +218,7 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)extern int is_pack_valid (struct packed_git *); +extern struct packed_git *find_sha1_pack (const unsigned char *sha1, +struct packed_git *packs); +#endif
packfile.c +13 -0
@@ -1783,3 +1783,16 @@ int is_pack_valid(struct packed_git *p)/* Force the pack to open to prove its valid. */ return !open_packed_git (p); } + +struct packed_git *find_sha1_pack (const unsigned char *sha1, +struct packed_git *packs) +{ +struct packed_git *p; + +for (p = packs; p; p = p->next) { +if (find_pack_entry_one (sha1, p)) +return p; + } +return NULL ; + +}
sha1_file.c +0 -13
@@ -1126,19 +1126,6 @@ static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)return 0 ; } -struct packed_git *find_sha1_pack (const unsigned char *sha1, -struct packed_git *packs) -{ -struct packed_git *p; - -for (p = packs; p; p = p->next) { -if (find_pack_entry_one (sha1, p)) -return p; - } -return NULL ; - -} -static int sha1_loose_object_info (const unsigned char *sha1,struct object_info *oi,int flags)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 22/25 pack: move find_pack_entry() and make it global
In reply to Jonathan TanThis function needs to be global as it is used by sha1_file.c and will be used by packfile.c.
--- pack.h | 2 ++ packfile.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ sha1_file.c | 53 ----------------------------------------------------- 3 files changed, 55 insertions(+), 53 deletions(-)
pack.h +2 -0
@@ -221,4 +221,6 @@ extern int is_pack_valid(struct packed_git *);extern struct packed_git *find_sha1_pack (const unsigned char *sha1,struct packed_git *packs); +extern int find_pack_entry (const unsigned char *sha1,struct pack_entry *e); +#endif
packfile.c +53 -0
56 lines not on this page · raw message
sha1_file.c +0 -53
59 lines not on this page · raw message
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 23/25 pack: move has_sha1_pack()
In reply to Jonathan Tan--- builtin/prune-packed.c | 1 + cache.h | 2 -- diff.c | 1 + pack.h | 2 ++ packfile.c | 6 ++++++ revision.c | 1 + sha1_file.c | 6 ------ 7 files changed, 11 insertions(+), 8 deletions(-)
builtin/prune-packed.c +1 -0
@@ -2,6 +2,7 @@#include "cache.h" #include "progress.h" #include "parse-options.h" +#include "pack.h" static const char *const prune_packed_usage[] = {N_ ("git prune-packed [-n | --dry-run] [-q | --quiet]" ),
cache.h +0 -2
@@ -1190,8 +1190,6 @@ extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned lextern int finalize_object_file (const char *tmpfile,const char *filename); -extern int has_sha1_pack (const unsigned char *sha1); -/* * Open the loose object at path, check its sha1, and return the contents, * type, and size. If the object is a blob, then "contents" may return NULL,
diff.c +1 -0
@@ -20,6 +20,7 @@#include "string-list.h" #include "argv-array.h" #include "graph.h" +#include "pack.h" #ifdef NO_FAST_WORKING_DIRECTORY #define FAST_WORKING_DIRECTORY 0
pack.h +2 -0
@@ -223,4 +223,6 @@ extern struct packed_git *find_sha1_pack(const unsigned char *sha1,extern int find_pack_entry (const unsigned char *sha1,struct pack_entry *e); +extern int has_sha1_pack (const unsigned char *sha1); +#endif
packfile.c +6 -0
@@ -1849,3 +1849,9 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e) }return 0 ; } + +int has_sha1_pack (const unsigned char *sha1) +{ +struct pack_entry e; +return find_pack_entry (sha1, &e); +}
revision.c +1 -0
@@ -19,6 +19,7 @@#include "dir.h" #include "cache-tree.h" #include "bisect.h" +#include "pack.h" volatile show_early_output_fn_t show_early_output;
sha1_file.c +0 -6
@@ -1629,12 +1629,6 @@ int has_pack_index(const unsigned char *sha1)return 1 ; } -int has_sha1_pack (const unsigned char *sha1) -{ -struct pack_entry e; -return find_pack_entry (sha1, &e); -} -int has_sha1_file_with_flags (const unsigned char *sha1,int flags) {if (!startup_info->have_repository)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 24/25 pack: move has_pack_index()
In reply to Jonathan Tan--- cache.h | 2 -- pack.h | 2 ++ packfile.c | 8 ++++++++ sha1_file.c | 8 -------- 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -2
@@ -1225,8 +1225,6 @@ extern int has_object_file_with_flags(const struct object_id *oid, int flags);*/ extern int has_loose_object_nonlocal (const unsigned char *sha1); -extern int has_pack_index (const unsigned char *sha1); -extern void assert_sha1_type (const unsigned char *sha1,enum object_type expect);/* Helper to check and "touch" a file */
pack.h +2 -0
@@ -225,4 +225,6 @@ extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);extern int has_sha1_pack (const unsigned char *sha1); +extern int has_pack_index (const unsigned char *sha1); +#endif
packfile.c +8 -0
@@ -1855,3 +1855,11 @@ int has_sha1_pack(const unsigned char *sha1)struct pack_entry e;return find_pack_entry (sha1, &e); } + +int has_pack_index (const unsigned char *sha1) +{ +struct stat st; +if (stat (sha1_pack_index_name (sha1), &st)) +return 0 ; +return 1 ; +}
sha1_file.c +0 -8
@@ -1621,14 +1621,6 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)return ret; } -int has_pack_index (const unsigned char *sha1) -{ -struct stat st; -if (stat (sha1_pack_index_name (sha1), &st)) -return 0 ; -return 1 ; -} -int has_sha1_file_with_flags (const unsigned char *sha1,int flags) {if (!startup_info->have_repository)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH v2 25/25 pack: move for_each_packed_object()
In reply to Jonathan Tan--- builtin/cat-file.c | 1 + cache.h | 7 +------ pack.h | 11 +++++++++++ packfile.c | 40 ++++++++++++++++++++++++++++++++++++++++ reachable.c | 1 + sha1_file.c | 40 ---------------------------------------- 6 files changed, 54 insertions(+), 46 deletions(-)
builtin/cat-file.c +1 -0
@@ -12,6 +12,7 @@#include "streaming.h" #include "tree-walk.h" #include "sha1-array.h" +#include "pack.h" struct batch_options {int enabled;
cache.h +1 -6
@@ -1660,17 +1660,12 @@ int for_each_loose_file_in_objdir_buf(struct strbuf *path, void *data);/* -* Iterate over loose and packed objects in both the local +* Iterate over loose objects in both the local * repository and any alternates repositories (unless the * LOCAL_ONLY flag is set). */ #define FOR_EACH_OBJECT_LOCAL_ONLY 0x1 -typedef int each_packed_object_fn (const struct object_id *oid, -struct packed_git *pack, -uint32_t pos, -void *data);extern int for_each_loose_object (each_loose_object_fn ,void *,unsigned flags); -extern int for_each_packed_object (each_packed_object_fn ,void *,unsigned flags);struct object_info {/* Request */
pack.h +11 -0
@@ -227,4 +227,15 @@ extern int has_sha1_pack(const unsigned char *sha1);extern int has_pack_index (const unsigned char *sha1); +/* +* Iterate over packed objects in both the local +* repository and any alternates repositories (unless the +* FOR_EACH_OBJECT_LOCAL_ONLY flag, defined in cache.h, is set). +*/ +typedef int each_packed_object_fn (const struct object_id *oid, +struct packed_git *pack, +uint32_t pos, +void *data); +extern int for_each_packed_object (each_packed_object_fn ,void *,unsigned flags); +#endif
packfile.c +40 -0
@@ -1863,3 +1863,43 @@ int has_pack_index(const unsigned char *sha1)return 0 ;return 1 ; } + +static int for_each_object_in_pack (struct packed_git *p,each_packed_object_fn cb,void *data) +{ +uint32_t i; +int r =0 ; + +for (i =0 ; i < p->num_objects; i++) { +struct object_id oid; + +if (!nth_packed_object_oid (&oid, p, i)) +return error ("unable to get sha1 of object %u in %s" , + i, p->pack_name); + + r =cb (&oid, p, i, data); +if (r) +break ; + } +return r; +} + +int for_each_packed_object (each_packed_object_fn cb,void *data,unsigned flags) +{ +struct packed_git *p; +int r =0 ; +int pack_errors =0 ; + +prepare_packed_git (); +for (p = packed_git; p; p = p->next) { +if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local) +continue ; +if (open_pack_index (p)) { + pack_errors =1 ; +continue ; + } + r =for_each_object_in_pack (p, cb, data); +if (r) +break ; + } +return r ? r : pack_errors; +}
reachable.c +1 -0
@@ -9,6 +9,7 @@#include "cache-tree.h" #include "progress.h" #include "list-objects.h" +#include "pack.h" struct connectivity_progress {struct progress *progress;
sha1_file.c +0 -40
@@ -2013,46 +2013,6 @@ int for_each_loose_object(each_loose_object_fn cb, void *data, unsigned flags)return foreach_alt_odb (loose_from_alt_odb, &alt); } -static int for_each_object_in_pack (struct packed_git *p,each_packed_object_fn cb,void *data) -{ -uint32_t i; -int r =0 ; - -for (i =0 ; i < p->num_objects; i++) { -struct object_id oid; - -if (!nth_packed_object_oid (&oid, p, i)) -return error ("unable to get sha1 of object %u in %s" , - i, p->pack_name); - - r =cb (&oid, p, i, data); -if (r) -break ; - } -return r; -} - -int for_each_packed_object (each_packed_object_fn cb,void *data,unsigned flags) -{ -struct packed_git *p; -int r =0 ; -int pack_errors =0 ; - -prepare_packed_git (); -for (p = packed_git; p; p = p->next) { -if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local) -continue ; -if (open_pack_index (p)) { - pack_errors =1 ; -continue ; - } - r =for_each_object_in_pack (p, cb, data); -if (r) -break ; - } -return r ? r : pack_errors; -} - static int check_stream_sha1(git_zstream *stream, const char *hdr, unsigned long size,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
Re: [PATCH v2 00/25] Move exported packfile funcs to its own file
In reply to Jonathan TanQuoted 16 lines · Jonathan Tan
Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like
freshen_packed_object()that are used only by non-pack-specific functions are not moved.In the end, 3 functions needed to be made global. They are
find_pack_entry(),mark_bad_packed_object(), andhas_packed_and_bad().Of the 3,
find_pack_entry()is probably legitimately promoted. But I think that the latter two functions needing to be accessed fromsha1_file.cpoints to a design that could be improved - they are only used whenpacked_object_info()detects corruption, and used for marking as bad and printing messages to the user respectively, whichpacked_object_info()should probably do itself. But I have not made this change in this patch set.
Quoted 3 lines
(Other than the 3 functions above, there are some variables and functions that are temporarily made global, but reduced back to static when the wide scope is no longer needed.)
I read through the patches yesterday and had no comment.
Thanks,
Stefan
Re: [PATCH v2 00/25] Move exported packfile funcs to its own file
In reply to Jonathan TanQuoted 4 lines · Jonathan Tan
Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like
freshen_packed_object()that are used only by non-pack-specific functions are not moved.
This will interfere with smaller changes and fixes we want to have early in the 'master' branch, so while I think it is a good idea to do something like this in the longer term, I'd have to ask you to either hold on or rebase this on them (you'll know what else you are conflicting with when you try to merge this to 'pu' yourself).
Thanks.
Re: [PATCH v2 00/25] Move exported packfile funcs to its own file
Quoted 7 lines · On Thu, 10 Aug 2017 14:19:59 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 4 lines · Jonathan Tan
Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like
freshen_packed_object()that are used only by non-pack-specific functions are not moved.
Quoted 7 lines
This will interfere with smaller changes and fixes we want to have early in the 'master' branch, so while I think it is a good idea to do something like this in the longer term, I'd have to ask you to either hold on or rebase this on them (you'll know what else you are conflicting with when you try to merge this to 'pu' yourself).
Thanks.
OK, I'll wait until you have updated the master branch, then I'll try to rebase on it.
Re: [PATCH v2 00/25] Move exported packfile funcs to its own file
Quoted 16 lines · Jonathan Tan
Quoted 13 lines · On Thu, 10 Aug 2017 14:19:59 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 4 lines · Jonathan Tan
Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like
freshen_packed_object()that are used only by non-pack-specific functions are not moved.This will interfere with smaller changes and fixes we want to have early in the 'master' branch, so while I think it is a good idea to do something like this in the longer term, I'd have to ask you to either hold on or rebase this on them (you'll know what else you are conflicting with when you try to merge this to 'pu' yourself).
Quoted 4 lines
Quoted 1 line
Thanks.
OK, I'll wait until you have updated the master branch, then I'll try to rebase on it.
That will take a few weeks, and I do not think we want you idling during that time ;-).
You'd need to double check, but I think the topics that cause trouble are rs/find-apck-entry-bisection and jk/drop-sha1-entry-pos; you can start from v2.14.1 and merge these topics on top and then build your change on top. That would allow you to start cooking before both of them graduate to 'master', as I expect they are both quick-to-next material. There might be other topics that interfere with what you are doing, but you can easily find out what they are if you do a trial merge to 'next' and 'pu' yourself.
Thanks.
cover non-move patches in preparation for packfile.c
Thanks, I'll work on that.
In the meantime, could these 2 patches be merged in (after review, of course) first? This will make the remaining set much easier to review, as you can check most of them with the new --color-moved functionality of diff.
Incidentally, after rebasing on "pu" and resolving the conflicts, I noticed some "zebra stripes" (from --color-moved) where I didn't expect time. I'll try to figure out that issue before I resend the rest of the patches (which will be after the relevant branches are merged to "next").
Jonathan Tan (2):
sha1_file: set whence in storage-specific info fn
sha1_file: remove read_packed_sha1()
sha1_file.c | 39 +++++++-------------------------------- 1 file changed, 7 insertions(+), 32 deletions(-)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
PATCH 1/2 sha1_file: set whence in storage-specific info fn
Move the setting of oi->whence to sha1_loose_object_info() and packed_object_info().
This allows sha1_object_info_extended() to not need to know about the delta base cache. This will be useful during a future refactoring in which packfile-related functions, including the handling of the delta base cache, will be moved to a separate file.
--- sha1_file.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)
sha1_file.c +6 -7
@@ -2444,6 +2444,9 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,hashclr (oi->delta_base_sha1); } + oi->whence =in_delta_base_cache (p, obj_offset) ?OI_DBCACHED : +OI_PACKED ; + out:unuse_pack (&w_curs);return type;
@@ -2973,6 +2976,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,if (oi->sizep == &size_scratch) oi->sizep =NULL ;strbuf_release (&hdrbuf); + oi->whence =OI_LOOSE ;return (status <0 ) ? status :0 ; }
@@ -3010,10 +3014,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,if (!find_pack_entry (real, &e)) {/* Most likely it's a loose object. */ -if (!sha1_loose_object_info (real, oi, flags)) { - oi->whence =OI_LOOSE ; +if (!sha1_loose_object_info (real, oi, flags))return 0 ; - }/* Not a loose object; someone else may have just packed it. */ if (flags &OBJECT_INFO_QUICK ) {
@@ -3036,10 +3038,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,if (rtype <0 ) {mark_bad_packed_object (e.p, real);return sha1_object_info_extended (real, oi,0 ); - }else if (in_delta_base_cache (e.p, e.offset)) { - oi->whence =OI_DBCACHED ; - }else { - oi->whence =OI_PACKED ; + }else if (oi->whence ==OI_PACKED ) { oi->u.packed.offset = e.offset; oi->u.packed.pack = e.p; oi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
Re: [PATCH 1/2] sha1_file: set whence in storage-specific info fn
Quoted 10 lines · Jonathan Tan
Move the setting of
oi->whencetosha1_loose_object_info()andpacked_object_info().This allows
sha1_object_info_extended()to not need to know about the delta base cache. This will be useful during a future refactoring in which packfile-related functions, including the handling of the delta base cache, will be moved to a separate file.Signed-off-by Jonathan Tan <[email protected]>---
Output from
git grep -E -e '(\.|->)whence'
makes me wonder if the oi->whence thing is a bit over-engineered, though. The only real user of this information is the streaming code, which wants to see if it can grab an undeltified deflated data directly out of the pack file (and if so from which packfile at what offset), or if it can open a loose object file and slurp deflated data out of it.
But that is totally outside the scope of this patch. This looks like a safe no-op conversion to me.
Thanks.
Quoted 49 lines
sha1_file.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-)sha1_file.c +6
-7@@ -2444,6 +2444,9 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,hashclr (oi->delta_base_sha1); } + oi->whence =in_delta_base_cache (p, obj_offset) ?OI_DBCACHED : +OI_PACKED ; + out:unuse_pack (&w_curs);return type;@@ -2973,6 +2976,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,if (oi->sizep == &size_scratch) oi->sizep =NULL ;strbuf_release (&hdrbuf); + oi->whence =OI_LOOSE ;return (status <0 ) ? status :0 ; }@@ -3010,10 +3014,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,if (!find_pack_entry (real, &e)) {/* Most likely it's a loose object. */ -if (!sha1_loose_object_info (real, oi, flags)) { - oi->whence =OI_LOOSE ; +if (!sha1_loose_object_info (real, oi, flags))return 0 ; - }/* Not a loose object; someone else may have just packed it. */ if (flags &OBJECT_INFO_QUICK ) {@@ -3036,10 +3038,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,if (rtype <0 ) {mark_bad_packed_object (e.p, real);return sha1_object_info_extended (real, oi,0 ); - }else if (in_delta_base_cache (e.p, e.offset)) { - oi->whence =OI_DBCACHED ; - }else { - oi->whence =OI_PACKED ; + }else if (oi->whence ==OI_PACKED ) { oi->u.packed.offset = e.offset; oi->u.packed.pack = e.p; oi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||
PATCH 2/2 sha1_file: remove read_packed_sha1()
In reply to Jonathan TanUse read_object() in its place instead. This avoids duplication of code.
This makes force_object_loose() slightly slower (because of a redundant check of loose object storage), but only in the error case.
--- sha1_file.c | 26 +------------------------- 1 file changed, 1 insertion(+), 25 deletions(-)
sha1_file.c +1 -25
@@ -3062,30 +3062,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)return type; } -static void *read_packed_sha1 (const unsigned char *sha1, -enum object_type *type,unsigned long *size) -{ -struct pack_entry e; -void *data; - -if (!find_pack_entry (sha1, &e)) -return NULL ; - data =cache_or_unpack_entry (e.p, e.offset, size, type); -if (!data) { -/* -* We're probably in deep shit, but let's try to fetch -* the required object anyway from another pack or loose. -* This should happen only in the presence of a corrupted -* pack, and is better than failing outright. -*/ -error ("failed to read object %s at offset %" PRIuMAX" from %s" , -sha1_to_hex (sha1), (uintmax_t )e.offset, e.p->pack_name); -mark_bad_packed_object (e.p, sha1); - data =read_object (sha1, type, size); - } -return data; -} -int pretend_sha1_file (void *buf,unsigned long len,enum object_type type,unsigned char *sha1) {
@@ -3468,7 +3444,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)if (has_loose_object (sha1))return 0 ; - buf =read_packed_sha1 (sha1, &type, &len); + buf =read_object (sha1, &type, &len);if (!buf)return error ("cannot read sha1_file for %s" ,sha1_to_hex (sha1)); hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %lu" ,typename (type), len) +1 ;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
Re: [PATCH 2/2] sha1_file: remove read_packed_sha1()
Quoted 9 lines · Jonathan Tan
Use
read_object()in its place instead. This avoids duplication of code.This makes
force_object_loose()slightly slower (because of a redundant check of loose object storage), but only in the error case.Signed-off-by Jonathan Tan <[email protected]>--- sha1_file.c | 26 +------------------------- 1 file changed, 1 insertion(+), 25 deletions(-)
The original code insisted on reading from pack and never from a loose object, because it knew it would return early when it found a loose version. Now we allow a loose one to appear in the middle of force_object_loose() operation and happily read from it when we do not see a pack entry for the object---presumably because we are racing with another simultanous repack process, or something?---and then write it out as a new (and identical) loose object, which would not do any harm.
So this is not strictly a no-op conversion; I have a gut feeling that it would make it more robust, not less, in the presence of another racing repack process, but I haven't really thought through race scenarios that may make difference in its behaviour.
Quoted 44 lines
sha1_file.c +1
-25@@ -3062,30 +3062,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)return type; } -static void *read_packed_sha1 (const unsigned char *sha1, -enum object_type *type,unsigned long *size) -{ -struct pack_entry e; -void *data; - -if (!find_pack_entry (sha1, &e)) -return NULL ; - data =cache_or_unpack_entry (e.p, e.offset, size, type); -if (!data) { -/* -* We're probably in deep shit, but let's try to fetch -* the required object anyway from another pack or loose. -* This should happen only in the presence of a corrupted -* pack, and is better than failing outright. -*/ -error ("failed to read object %s at offset %" PRIuMAX" from %s" , -sha1_to_hex (sha1), (uintmax_t )e.offset, e.p->pack_name); -mark_bad_packed_object (e.p, sha1); - data =read_object (sha1, type, size); - } -return data; -} -int pretend_sha1_file (void *buf,unsigned long len,enum object_type type,unsigned char *sha1) {@@ -3468,7 +3444,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)if (has_loose_object (sha1))return 0 ; - buf =read_packed_sha1 (sha1, &type, &len); + buf =read_object (sha1, &type, &len);if (!buf)return error ("cannot read sha1_file for %s" ,sha1_to_hex (sha1)); hdrlen =xsnprintf (hdr,sizeof (hdr),"%s %lu" ,typename (type), len) +1 ;
Re: [PATCH v2 00/25] Move exported packfile funcs to its own file
In reply to Jonathan TanQuoted 16 lines · Jonathan Tan
Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like
freshen_packed_object()that are used only by non-pack-specific functions are not moved.In the end, 3 functions needed to be made global. They are
find_pack_entry(),mark_bad_packed_object(), andhas_packed_and_bad().Of the 3,
find_pack_entry()is probably legitimately promoted. But I think that the latter two functions needing to be accessed fromsha1_file.cpoints to a design that could be improved - they are only used whenpacked_object_info()detects corruption, and used for marking as bad and printing messages to the user respectively, whichpacked_object_info()should probably do itself. But I have not made this change in this patch set.
Quoted 4 lines
(Other than the 3 functions above, there are some variables and functions that are temporarily made global, but reduced back to static when the wide scope is no longer needed.)
Nice to see the pack file functions being refactored out. I looked at the end result and it looked good to me.
Do you have the energy to do a similar refactoring for the remaining public functions residing in sha1_file.c? Perhaps a new sha1_file.h? It would be nice to get more things out of cache.h. :)
Re: [PATCH v2 00/25] Move exported packfile funcs to its own file
Quoted 2 lines · On Fri, 11 Aug 2017 15:41:28 -0400 Ben Peart <[email protected]> wrote:
Nice to see the pack file functions being refactored out. I looked at the end result and it looked good to me.
Thanks.
Quoted 3 lines
Do you have the energy to do a similar refactoring for the remaining public functions residing in
sha1_file.c? Perhaps a newsha1_file.h? It would be nice to get more things out ofcache.h. :)
I agree that that would be desirable, but for now I'll leave that for someone else to do :-)
v3 cover Move exported packfile funcs to its own file
In reply to Jonathan TanQuoted 8 lines
You'd need to double check, but I think the topics that cause trouble are rs/find-apck-entry-bisection and jk/drop-sha1-entry-pos; you can start from v2.14.1 and merge these topics on top and then build your change on top. That would allow you to start cooking before both of them graduate to 'master', as I expect they are both quick-to-next material. There might be other topics that interfere with what you are doing, but you can easily find out what they are if you do a trial merge to 'next' and 'pu' yourself.
OK - in addition to the 2 you mentioned, I have found some others (likely added after you wrote that). The complete list is:
- rs/find-pack-entry-bisection
- jk/drop-sha1-entry-pos
- jt/sha1-file-cleanup (formerly part of this set)
- mk/use-size-t-in-zlib
- rs/unpack-entry-leakfix
I have merged all of these and rebased my patches on top.
Other changes:
- Used packfile.h instead of pack.h (following most people's preference)
- Ensured that I added functions to packfile.h retaining the order they were originally in, so that if you run "git diff <base> --color-moved --patience", there are much fewer zebra stripes
The merge base commit can be accessed online [1], if you need it.
[1] https://github.com/jonathantanmy/git/commits/packmigrate
Jonathan Tan (23):
pack: move pack name-related functions
pack: move static state variables
pack: move pack_report()
pack: move open_pack_index(), parse_pack_index()
pack: move release_pack_memory()
pack: move pack-closing functions
pack: move use_pack()
pack: move unuse_pack()
pack: move add_packed_git()
pack: move install_packed_git()
pack: move {,re}prepare_packed_git and approximate_object_count
pack: move unpack_object_header_buffer()
pack: move get_size_from_delta()
pack: move unpack_object_header()
pack: move clear_delta_base_cache(), packed_object_info(),
unpack_entry()
pack: move nth_packed_object_{sha1,oid}
pack: move check_pack_index_ptr(), nth_packed_object_offset()
pack: move find_pack_entry_one(), is_pack_valid()
pack: move find_sha1_pack()
pack: move find_pack_entry() and make it global
pack: move has_sha1_pack()
pack: move has_pack_index()
pack: move for_each_packed_object()
Makefile | 1 + builtin/am.c | 1 + builtin/cat-file.c | 1 + builtin/clone.c | 1 + builtin/count-objects.c | 1 + builtin/fetch.c | 1 + builtin/fsck.c | 1 + builtin/gc.c | 1 + builtin/index-pack.c | 1 + builtin/merge.c | 1 + builtin/pack-objects.c | 1 + builtin/pack-redundant.c | 1 + builtin/prune-packed.c | 1 + builtin/receive-pack.c | 1 + bulk-checkin.c | 1 + cache.h | 122 +-- connected.c | 1 + diff.c | 1 + fast-import.c | 1 + fetch-pack.c | 1 + git-compat-util.h | 2 - http-backend.c | 1 + http-push.c | 1 + http-walker.c | 1 + http.c | 1 + outgoing/packfile.h | 0 pack-bitmap.c | 1 + pack-check.c | 1 + packfile.c | 1896 +++++++++++++++++++++++++++++++++++ packfile.h | 138 +++ path.c | 1 + reachable.c | 1 + revision.c | 1 + server-info.c | 1 + sha1_file.c | 2452 ++++++---------------------------------------- sha1_name.c | 1 + streaming.c | 1 + 37 files changed, 2354 insertions(+), 2287 deletions(-) create mode 100644 outgoing/packfile.h create mode 100644 packfile.c create mode 100644 packfile.h
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 1/23 pack: move pack name-related functions
Currently, sha1_file.c and cache.h contain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.
Create a new file, packfile.c, to hold all packfile-related functions currently in sha1_file.c. It has a corresponding header packfile.h.
In this commit, the pack name-related functions are moved. Subsequent commits will move the other functions.
--- Makefile | 1 + builtin/index-pack.c | 1 + builtin/pack-redundant.c | 1 + cache.h | 23 ----------------------- fast-import.c | 1 + http.c | 1 + outgoing/packfile.h | 0 packfile.c | 23 +++++++++++++++++++++++ packfile.h | 27 +++++++++++++++++++++++++++ sha1_file.c | 23 +---------------------- 10 files changed, 56 insertions(+), 45 deletions(-) create mode 100644 outgoing/packfile.h create mode 100644 packfile.c create mode 100644 packfile.h
Makefile +1 -0
@@ -816,6 +816,7 @@ LIB_OBJS += notes-merge.o LIB_OBJS += notes-utils.o LIB_OBJS += object.o LIB_OBJS += oidset.o +LIB_OBJS += packfile.o LIB_OBJS += pack-bitmap.o LIB_OBJS += pack-bitmap-write.o LIB_OBJS += pack-check.o
builtin/index-pack.c +1 -0
@@ -12,6 +12,7 @@#include "exec_cmd.h" #include "streaming.h" #include "thread-utils.h" +#include "packfile.h" static const char index_pack_usage[] ="git index-pack [-v] [-o <index-file>] [--keep | --keep=<msg>] [--verify] [--strict] (<pack-file> | --stdin [--fix-thin] [<pack-file>])" ;
builtin/pack-redundant.c +1 -0
@@ -7,6 +7,7 @@*/ #include "builtin.h" +#include "packfile.h" #define BLKSIZE 512
cache.h +0 -23
@@ -902,20 +902,6 @@ extern void check_repository_format(void);*/ extern const char *sha1_file_name (const unsigned char *sha1); -/* -* Return the name of the (local) packfile with the specified sha1 in -* its name. The return value is a pointer to memory that is -* overwritten each time this function is called. -*/ -extern char *sha1_pack_name (const unsigned char *sha1); - -/* -* Return the name of the (local) pack index file with the specified -* sha1 in its name. The return value is a pointer to memory that is -* overwritten each time this function is called. -*/ -extern char *sha1_pack_index_name (const unsigned char *sha1); -/* * Return an abbreviated sha1 unique within this repository's object database. * The result will be at least `len` characters long, and will be NUL
@@ -1656,15 +1642,6 @@ extern void pack_report(void);*/ extern int odb_mkstemp (struct strbuf *template,const char *pattern); -/* -* Generate the filename to be used for a pack file with checksum "sha1" and -* extension "ext". The result is written into the strbuf "buf", overwriting -* any existing contents. A pointer to buf->buf is returned as a convenience. -* -* Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" -*/ -extern char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); -/* * Create a pack .keep file named "name" (which should generally be the output * of odb_pack_name). Returns a file descriptor opened for writing, or -1 on
fast-import.c +1 -0
@@ -167,6 +167,7 @@ Format of STDIN stream:#include "quote.h" #include "dir.h" #include "run-command.h" +#include "packfile.h" #define PACK_ID_BITS 16#define MAX_PACK_ID ((1<<PACK_ID_BITS)-1)
http.c +1 -0
@@ -11,6 +11,7 @@#include "pkt-line.h" #include "gettext.h" #include "transport.h" +#include "packfile.h" static struct trace_key trace_curl =TRACE_KEY_INIT (CURL );#if LIBCURL_VERSION_NUM >=0x070a08
outgoing/packfile.h new file +0 -0
packfile.c new file +23 -0
@@ -0,0 +1,23 @@ +#include "cache.h" + +char *odb_pack_name (struct strbuf *buf, +const unsigned char *sha1, +const char *ext) +{ +strbuf_reset (buf); +strbuf_addf (buf,"%s/pack/pack-%s.%s" ,get_object_directory (), +sha1_to_hex (sha1), ext); +return buf->buf; +} + +char *sha1_pack_name (const unsigned char *sha1) +{ +static struct strbuf buf =STRBUF_INIT ; +return odb_pack_name (&buf, sha1,"pack" ); +} + +char *sha1_pack_index_name (const unsigned char *sha1) +{ +static struct strbuf buf =STRBUF_INIT ; +return odb_pack_name (&buf, sha1,"idx" ); +}
packfile.h new file +27 -0
@@ -0,0 +1,27 @@ +#ifndef PACKFILE_H +#define PACKFILE_H + +/* +* Generate the filename to be used for a pack file with checksum "sha1" and +* extension "ext". The result is written into the strbuf "buf", overwriting +* any existing contents. A pointer to buf->buf is returned as a convenience. +* +* Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx" +*/ +extern char *odb_pack_name (struct strbuf *buf,const unsigned char *sha1,const char *ext); + +/* +* Return the name of the (local) packfile with the specified sha1 in +* its name. The return value is a pointer to memory that is +* overwritten each time this function is called. +*/ +extern char *sha1_pack_name (const unsigned char *sha1); + +/* +* Return the name of the (local) pack index file with the specified +* sha1 in its name. The return value is a pointer to memory that is +* overwritten each time this function is called. +*/ +extern char *sha1_pack_index_name (const unsigned char *sha1); + +#endif
sha1_file.c +1 -22
@@ -28,6 +28,7 @@#include "list.h" #include "mergesort.h" #include "quote.h" +#include "packfile.h" #define SZ_FMT PRIuMAXstatic inline uintmax_t sz_fmt (size_t s) {return s; }
@@ -278,28 +279,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,return buf->buf; } -char *odb_pack_name (struct strbuf *buf, -const unsigned char *sha1, -const char *ext) -{ -strbuf_reset (buf); -strbuf_addf (buf,"%s/pack/pack-%s.%s" ,get_object_directory (), -sha1_to_hex (sha1), ext); -return buf->buf; -} - -char *sha1_pack_name (const unsigned char *sha1) -{ -static struct strbuf buf =STRBUF_INIT ; -return odb_pack_name (&buf, sha1,"pack" ); -} - -char *sha1_pack_index_name (const unsigned char *sha1) -{ -static struct strbuf buf =STRBUF_INIT ; -return odb_pack_name (&buf, sha1,"idx" ); -} -struct alternate_object_database *alt_odb_list;static struct alternate_object_database **alt_odb_tail;
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 2/23 pack: move static state variables
In reply to Jonathan Tansha1_file.c declares some static variables that store packfile-related state. Move them to packfile.c.
They are temporarily made global, but subsequent commits will restore their scope back to static.
--- packfile.c | 14 ++++++++++++++ packfile.h | 9 +++++++++ sha1_file.c | 13 ------------- 3 files changed, 23 insertions(+), 13 deletions(-)
packfile.c +14 -0
@@ -1,4 +1,5 @@#include "cache.h" +#include "mru.h" char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1,
@@ -21,3 +22,16 @@ char *sha1_pack_index_name(const unsigned char *sha1)static struct strbuf buf =STRBUF_INIT ;return odb_pack_name (&buf, sha1,"idx" ); } + +unsigned int pack_used_ctr; +unsigned int pack_mmap_calls; +unsigned int peak_pack_open_windows; +unsigned int pack_open_windows; +unsigned int pack_open_fds; +unsigned int pack_max_fds; +size_t peak_pack_mapped; +size_t pack_mapped; +struct packed_git *packed_git; + +static struct mru packed_git_mru_storage; +struct mru *packed_git_mru = &packed_git_mru_storage;
packfile.h +9 -0
@@ -24,4 +24,13 @@ extern char *sha1_pack_name(const unsigned char *sha1);*/ extern char *sha1_pack_index_name (const unsigned char *sha1); +extern unsigned int pack_used_ctr; +extern unsigned int pack_mmap_calls; +extern unsigned int peak_pack_open_windows; +extern unsigned int pack_open_windows; +extern unsigned int pack_open_fds; +extern unsigned int pack_max_fds; +extern size_t peak_pack_mapped; +extern size_t pack_mapped; +#endif
sha1_file.c +0 -13
@@ -683,19 +683,6 @@ static int has_loose_object(const unsigned char *sha1)return check_and_freshen (sha1,0 ); } -static unsigned int pack_used_ctr; -static unsigned int pack_mmap_calls; -static unsigned int peak_pack_open_windows; -static unsigned int pack_open_windows; -static unsigned int pack_open_fds; -static unsigned int pack_max_fds; -static size_t peak_pack_mapped; -static size_t pack_mapped; -struct packed_git *packed_git; - -static struct mru packed_git_mru_storage; -struct mru *packed_git_mru = &packed_git_mru_storage; -void pack_report (void ) { fprintf(stderr,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 3/23 pack: move pack_report()
In reply to Jonathan Tan--- cache.h | 2 -- packfile.c | 24 ++++++++++++++++++++++++ packfile.h | 2 ++ sha1_file.c | 24 ------------------------ 4 files changed, 26 insertions(+), 26 deletions(-)
cache.h +0 -2
@@ -1632,8 +1632,6 @@ unsigned long approximate_object_count(void);extern struct packed_git *find_sha1_pack (const unsigned char *sha1,struct packed_git *packs); -extern void pack_report (void ); -/* * Create a temporary file rooted in the object database directory, or * die on failure. The filename is taken from "pattern", which should have the
packfile.c +24 -0
@@ -35,3 +35,27 @@ struct packed_git *packed_git;static struct mru packed_git_mru_storage;struct mru *packed_git_mru = &packed_git_mru_storage; + +#define SZ_FMT PRIuMAX +static inline uintmax_t sz_fmt (size_t s) {return s; } + +void pack_report (void ) +{ +fprintf (stderr, +"pack_report: getpagesize() = %10" SZ_FMT "\n" +"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" +"pack_report: core.packedGitLimit = %10" SZ_FMT "\n" , +sz_fmt (getpagesize ()), +sz_fmt (packed_git_window_size), +sz_fmt (packed_git_limit)); +fprintf (stderr, +"pack_report: pack_used_ctr = %10u\n" +"pack_report: pack_mmap_calls = %10u\n" +"pack_report: pack_open_windows = %10u / %10u\n" +"pack_report: pack_mapped = " +"%10" SZ_FMT " / %10" SZ_FMT "\n" , + pack_used_ctr, + pack_mmap_calls, + pack_open_windows, peak_pack_open_windows, +sz_fmt (pack_mapped),sz_fmt (peak_pack_mapped)); +}
packfile.h +2 -0
@@ -33,4 +33,6 @@ extern unsigned int pack_max_fds;extern size_t peak_pack_mapped;extern size_t pack_mapped; +extern void pack_report (void ); +#endif
sha1_file.c +0 -24
@@ -30,9 +30,6 @@#include "quote.h" #include "packfile.h" -#define SZ_FMT PRIuMAX -static inline uintmax_t sz_fmt (size_t s) {return s; } -const unsigned char null_sha1[20 ];const struct object_id null_oid;const struct object_id empty_tree_oid = {
@@ -683,27 +680,6 @@ static int has_loose_object(const unsigned char *sha1)return check_and_freshen (sha1,0 ); } -void pack_report (void ) -{ -fprintf (stderr, -"pack_report: getpagesize() = %10" SZ_FMT "\n" -"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n" -"pack_report: core.packedGitLimit = %10" SZ_FMT "\n" , -sz_fmt (getpagesize ()), -sz_fmt (packed_git_window_size), -sz_fmt (packed_git_limit)); -fprintf (stderr, -"pack_report: pack_used_ctr = %10u\n" -"pack_report: pack_mmap_calls = %10u\n" -"pack_report: pack_open_windows = %10u / %10u\n" -"pack_report: pack_mapped = " -"%10" SZ_FMT " / %10" SZ_FMT "\n" , - pack_used_ctr, - pack_mmap_calls, - pack_open_windows, peak_pack_open_windows, -sz_fmt (pack_mapped),sz_fmt (peak_pack_mapped)); -} -/* * Open and mmap the index file at path, perform a couple of * consistency checks, then record its information to p. Return 0 on
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 4/23 pack: move open_pack_index(), parse_pack_index()
In reply to Jonathan Tanalloc_packed_git() in packfile.c is duplicated from sha1_file.c. In a subsequent commit, alloc_packed_git() will be removed from sha1_file.c.
--- builtin/count-objects.c | 1 + builtin/fsck.c | 1 + builtin/pack-objects.c | 1 + cache.h | 8 --- pack-bitmap.c | 1 + pack-check.c | 1 + packfile.c | 149 ++++++++++++++++++++++++++++++++++++++++++++++++ packfile.h | 8 +++ sha1_file.c | 140 --------------------------------------------- sha1_name.c | 1 + 10 files changed, 163 insertions(+), 148 deletions(-)
builtin/count-objects.c +1 -0
@@ -10,6 +10,7 @@#include "builtin.h" #include "parse-options.h" #include "quote.h" +#include "packfile.h" static unsigned long garbage;static off_t size_garbage;
builtin/fsck.c +1 -0
@@ -15,6 +15,7 @@#include "progress.h" #include "streaming.h" #include "decorate.h" +#include "packfile.h" #define REACHABLE 0x0001#define SEEN 0x0002
builtin/pack-objects.c +1 -0
@@ -25,6 +25,7 @@#include "sha1-array.h" #include "argv-array.h" #include "mru.h" +#include "packfile.h" static const char *pack_usage[] = {N_ ("git pack-objects --stdout [<options>...] [< <ref-list> | < <object-list>]" ),
cache.h +0 -8
@@ -1611,8 +1611,6 @@ struct pack_entry {struct packed_git *p; }; -extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); -/* A hook to report invalid files in pack directory */ #define PACKDIR_FILE_PACK 1#define PACKDIR_FILE_IDX 2
@@ -1647,12 +1645,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* mmap the index file for the specified packfile (if it is not -* already mmapped). Return 0 on success. -*/ -extern int open_pack_index (struct packed_git *); -/* * munmap the index file for the specified packfile (if it is * currently mmapped).
pack-bitmap.c +1 -0
@@ -9,6 +9,7 @@#include "pack-bitmap.h" #include "pack-revindex.h" #include "pack-objects.h" +#include "packfile.h" /* * An entry on the bitmap index, representing the bitmap for a given
pack-check.c +1 -0
@@ -2,6 +2,7 @@#include "pack.h" #include "pack-revindex.h" #include "progress.h" +#include "packfile.h" struct idx_entry {off_t offset;
packfile.c +149 -0
157 lines not on this page · raw message
packfile.h +8 -0
@@ -33,6 +33,14 @@ extern unsigned int pack_max_fds;extern size_t peak_pack_mapped;extern size_t pack_mapped; +extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +extern void pack_report (void ); +/* +* mmap the index file for the specified packfile (if it is not +* already mmapped). Return 0 on success. +*/ +extern int open_pack_index (struct packed_git *); +#endif
sha1_file.c +0 -140
152 lines not on this page · raw message
sha1_name.c +1 -0
@@ -9,6 +9,7 @@#include "remote.h" #include "dir.h" #include "sha1-array.h" +#include "packfile.h" static int get_sha1_oneline (const char *,unsigned char *,struct commit_list *);
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 5/23 pack: move release_pack_memory()
In reply to Jonathan TanThe function unuse_one_window() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.
--- git-compat-util.h | 2 -- packfile.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ packfile.h | 4 ++++ sha1_file.c | 49 ------------------------------------------------- 4 files changed, 53 insertions(+), 51 deletions(-)
git-compat-util.h +0 -2
@@ -749,8 +749,6 @@ const char *inet_ntop(int af, const void *src, char *dst, size_t size);extern int git_atexit (void (*handler)(void ));#endif -extern void release_pack_memory (size_t ); -typedef void (*try_to_free_t )(size_t );extern try_to_free_t set_try_to_free_routine (try_to_free_t );
packfile.c +49 -0
52 lines not on this page · raw message
packfile.h +4 -0
@@ -43,4 +43,8 @@ extern void pack_report(void);*/ extern int open_pack_index (struct packed_git *); +extern int unuse_one_window (struct packed_git *current); + +extern void release_pack_memory (size_t ); +#endif
sha1_file.c +0 -49
55 lines not on this page · raw message
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 6/23 pack: move pack-closing functions
In reply to Jonathan TanThe function close_pack_fd() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.
--- builtin/am.c | 1 + builtin/clone.c | 1 + builtin/fetch.c | 1 + builtin/merge.c | 1 + builtin/receive-pack.c | 1 + cache.h | 8 -------- packfile.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++ packfile.h | 11 ++++++++++ sha1_file.c | 55 -------------------------------------------------- 9 files changed, 70 insertions(+), 63 deletions(-)
builtin/am.c +1 -0
@@ -31,6 +31,7 @@#include "mailinfo.h" #include "apply.h" #include "string-list.h" +#include "packfile.h" /** * Returns 1 if the file is empty or does not exist, 0 otherwise.
builtin/clone.c +1 -0
@@ -25,6 +25,7 @@#include "remote.h" #include "run-command.h" #include "connected.h" +#include "packfile.h" /* * Overall FIXMEs:
builtin/fetch.c +1 -0
@@ -17,6 +17,7 @@#include "connected.h" #include "argv-array.h" #include "utf8.h" +#include "packfile.h" static const char *const builtin_fetch_usage[] = {N_ ("git fetch [<options>] [<repository> [<refspec>...]]" ),
builtin/merge.c +1 -0
@@ -32,6 +32,7 @@#include "gpg-interface.h" #include "sequencer.h" #include "string-list.h" +#include "packfile.h" #define DEFAULT_TWOHEAD (1<<0)#define DEFAULT_OCTOPUS (1<<1)
builtin/receive-pack.c +1 -0
@@ -23,6 +23,7 @@#include "fsck.h" #include "tmp-objdir.h" #include "oidset.h" +#include "packfile.h" static const char *const receive_pack_usage[] = {N_ ("git receive-pack <git-dir>" ),
cache.h +0 -8
@@ -1645,15 +1645,7 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* munmap the index file for the specified packfile (if it is -* currently mmapped). -*/ -extern void close_pack_index (struct packed_git *); -extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,size_t *); -extern void close_pack_windows (struct packed_git *); -extern void close_all_packs (void );extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
packfile.c +54 -0
57 lines not on this page · raw message
packfile.h +11 -0
@@ -43,6 +43,17 @@ extern void pack_report(void);*/ extern int open_pack_index (struct packed_git *); +/* +* munmap the index file for the specified packfile (if it is +* currently mmapped). +*/ +extern void close_pack_index (struct packed_git *); + +extern void close_pack_windows (struct packed_git *); +extern void close_all_packs (void ); + +extern int close_pack_fd (struct packed_git *); +extern int unuse_one_window (struct packed_git *current);extern void release_pack_memory (size_t );
sha1_file.c +0 -55
67 lines not on this page · raw message
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 7/23 pack: move use_pack()
In reply to Jonathan TanThe function open_packed_git() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.
--- cache.h | 1 - packfile.c | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- packfile.h | 14 +-- sha1_file.c | 285 -------------------------------------------------------- streaming.c | 1 + 5 files changed, 298 insertions(+), 306 deletions(-)
cache.h +0 -1
@@ -1645,7 +1645,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,size_t *);extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
packfile.c +294 -9
325 lines not on this page · raw message
packfile.h +3 -11
@@ -24,14 +24,7 @@ extern char *sha1_pack_name(const unsigned char *sha1);*/ extern char *sha1_pack_index_name (const unsigned char *sha1); -extern unsigned int pack_used_ctr; -extern unsigned int pack_mmap_calls; -extern unsigned int peak_pack_open_windows; -extern unsigned int pack_open_windows;extern unsigned int pack_open_fds; -extern unsigned int pack_max_fds; -extern size_t peak_pack_mapped; -extern size_t pack_mapped;extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path);
@@ -49,13 +42,12 @@ extern int open_pack_index(struct packed_git *);*/ extern void close_pack_index (struct packed_git *); +extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,size_t *);extern void close_pack_windows (struct packed_git *);extern void close_all_packs (void ); -extern int close_pack_fd (struct packed_git *); - -extern int unuse_one_window (struct packed_git *current); -extern void release_pack_memory (size_t ); +extern int open_packed_git (struct packed_git *p); +#endif
sha1_file.c +0 -285
297 lines not on this page · raw message
streaming.c +1 -0
@@ -3,6 +3,7 @@*/ #include "cache.h" #include "streaming.h" +#include "packfile.h" enum input_source { stream_error =-1 ,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 8/23 pack: move unuse_pack()
In reply to Jonathan Tan--- cache.h | 1 - packfile.c | 9 +++++++++ packfile.h | 1 + sha1_file.c | 9 --------- 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -1
@@ -1645,7 +1645,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -extern void unuse_pack (struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);
packfile.c +9 -0
@@ -596,3 +596,12 @@ unsigned char *use_pack(struct packed_git *p, *left = win->len -xsize_t (offset);return win->base + offset; } + +void unuse_pack (struct pack_window **w_cursor) +{ +struct pack_window *w = *w_cursor; +if (w) { + w->inuse_cnt--; + *w_cursor =NULL ; + } +}
packfile.h +1 -0
@@ -45,6 +45,7 @@ extern void close_pack_index(struct packed_git *);extern unsigned char *use_pack (struct packed_git *,struct pack_window **,off_t ,size_t *);extern void close_pack_windows (struct packed_git *);extern void close_all_packs (void ); +extern void unuse_pack (struct pack_window **);extern void release_pack_memory (size_t );
sha1_file.c +0 -9
@@ -718,15 +718,6 @@ void *xmmap(void *start, size_t length,return ret; } -void unuse_pack (struct pack_window **w_cursor) -{ -struct pack_window *w = *w_cursor; -if (w) { - w->inuse_cnt--; - *w_cursor =NULL ; - } -} -static struct packed_git *alloc_packed_git (int extra) {struct packed_git *p =xmalloc (st_add (sizeof (*p), extra));
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 9/23 pack: move add_packed_git()
In reply to Jonathan Tan--- cache.h | 1 - connected.c | 1 + packfile.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ packfile.h | 1 + sha1_file.c | 61 ------------------------------------------------------------- 5 files changed, 55 insertions(+), 62 deletions(-)
cache.h +0 -1
@@ -1646,7 +1646,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);extern int odb_pack_keep (const char *name);extern void clear_delta_base_cache (void ); -extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);/* * Make sure that a pointer access into an mmap'd index file is within bounds,
connected.c +1 -0
@@ -3,6 +3,7 @@#include "sigchain.h" #include "connected.h" #include "transport.h" +#include "pack.h" /* * If we feed all the commits we want to verify to this command
packfile.c +53 -0
56 lines not on this page · raw message
packfile.h +1 -0
@@ -46,6 +46,7 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_textern void close_pack_windows (struct packed_git *);extern void close_all_packs (void );extern void unuse_pack (struct pack_window **); +extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);extern void release_pack_memory (size_t );
sha1_file.c +0 -61
67 lines not on this page · raw message
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 10/23 pack: move install_packed_git()
In reply to Jonathan Tan--- cache.h | 1 - packfile.c | 11 ++++++++++- packfile.h | 2 ++ sha1_file.c | 9 --------- 4 files changed, 12 insertions(+), 11 deletions(-)
cache.h +0 -1
@@ -1619,7 +1619,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);extern void prepare_packed_git (void );extern void reprepare_packed_git (void ); -extern void install_packed_git (struct packed_git *pack);/* * Give a rough count of objects in the repository. This sacrifices accuracy
packfile.c +10 -1
@@ -28,7 +28,7 @@ static unsigned int pack_used_ctr;static unsigned int pack_mmap_calls;static unsigned int peak_pack_open_windows;static unsigned int pack_open_windows; -unsigned int pack_open_fds; +static unsigned int pack_open_fds;static unsigned int pack_max_fds;static size_t peak_pack_mapped;static size_t pack_mapped;
@@ -658,3 +658,12 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)hashclr (p->sha1);return p; } + +void install_packed_git (struct packed_git *pack) +{ +if (pack->pack_fd !=-1 ) + pack_open_fds++; + + pack->next = packed_git; + packed_git = pack; +}
packfile.h +2 -0
@@ -28,6 +28,8 @@ extern unsigned int pack_open_fds;extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +extern void install_packed_git (struct packed_git *pack); +extern void pack_report (void );/*
sha1_file.c +0 -9
@@ -718,15 +718,6 @@ void *xmmap(void *start, size_t length,return ret; } -void install_packed_git (struct packed_git *pack) -{ -if (pack->pack_fd !=-1 ) - pack_open_fds++; - - pack->next = packed_git; - packed_git = pack; -} -void (*report_garbage)(unsigned seen_bits,const char *path); static void report_helper(const struct string_list *list,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 11/23 pack: move {,re}prepare_packed_git and approximate_object_count
In reply to Jonathan Tan--- builtin/gc.c | 1 + bulk-checkin.c | 1 + cache.h | 15 ---- connected.c | 2 +- fetch-pack.c | 1 + http-backend.c | 1 + packfile.c | 217 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ packfile.h | 16 ++++- path.c | 1 + server-info.c | 1 + sha1_file.c | 214 -------------------------------------------------------- 11 files changed, 238 insertions(+), 232 deletions(-)
builtin/gc.c +1 -0
@@ -19,6 +19,7 @@#include "sigchain.h" #include "argv-array.h" #include "commit.h" +#include "packfile.h" #define FAILED_RUN "failed to run %s"
bulk-checkin.c +1 -0
@@ -6,6 +6,7 @@#include "csum-file.h" #include "pack.h" #include "strbuf.h" +#include "packfile.h" static struct bulk_checkin_state { unsigned plugged:1;
cache.h +0 -15
@@ -1611,21 +1611,6 @@ struct pack_entry {struct packed_git *p; }; -/* A hook to report invalid files in pack directory */ -#define PACKDIR_FILE_PACK 1 -#define PACKDIR_FILE_IDX 2 -#define PACKDIR_FILE_GARBAGE 4 -extern void (*report_garbage)(unsigned seen_bits,const char *path); - -extern void prepare_packed_git (void ); -extern void reprepare_packed_git (void ); - -/* -* Give a rough count of objects in the repository. This sacrifices accuracy -* for speed. -*/ -unsigned long approximate_object_count (void ); -extern struct packed_git *find_sha1_pack (const unsigned char *sha1,struct packed_git *packs);
connected.c +1 -1
@@ -3,7 +3,7 @@#include "sigchain.h" #include "connected.h" #include "transport.h" -#include "pack.h" +#include "packfile.h" /* * If we feed all the commits we want to verify to this command
fetch-pack.c +1 -0
@@ -17,6 +17,7 @@#include "prio-queue.h" #include "sha1-array.h" #include "oidset.h" +#include "packfile.h" static int transfer_unpack_limit =-1 ;static int fetch_unpack_limit =-1 ;
http-backend.c +1 -0
@@ -9,6 +9,7 @@#include "string-list.h" #include "url.h" #include "argv-array.h" +#include "packfile.h" static const char content_type[] ="Content-Type" ;static const char content_length[] ="Content-Length" ;
packfile.c +217 -0
226 lines not on this page · raw message
packfile.h +14 -2
@@ -24,12 +24,24 @@ extern char *sha1_pack_name(const unsigned char *sha1);*/ extern char *sha1_pack_index_name (const unsigned char *sha1); -extern unsigned int pack_open_fds; -extern struct packed_git *parse_pack_index (unsigned char *sha1,const char *idx_path); +/* A hook to report invalid files in pack directory */ +#define PACKDIR_FILE_PACK 1 +#define PACKDIR_FILE_IDX 2 +#define PACKDIR_FILE_GARBAGE 4 +extern void (*report_garbage)(unsigned seen_bits,const char *path); + +extern void prepare_packed_git (void ); +extern void reprepare_packed_git (void );extern void install_packed_git (struct packed_git *pack); +/* +* Give a rough count of objects in the repository. This sacrifices accuracy +* for speed. +*/ +unsigned long approximate_object_count (void ); +extern void pack_report (void );/*
path.c +1 -0
@@ -9,6 +9,7 @@#include "worktree.h" #include "submodule-config.h" #include "path.h" +#include "packfile.h" static int get_st_mode_bits (const char *path,int *mode) {
server-info.c +1 -0
@@ -3,6 +3,7 @@#include "object.h" #include "commit.h" #include "tag.h" +#include "packfile.h" /* * Create the file "path" by writing to a temporary file and renaming
sha1_file.c +0 -214
220 lines not on this page · raw message
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 12/23 pack: move unpack_object_header_buffer()
In reply to Jonathan Tan--- cache.h | 1 - packfile.c | 25 +++++++++++++++++++++++++ packfile.h | 2 ++ sha1_file.c | 25 ------------------------- 4 files changed, 27 insertions(+), 26 deletions(-)
cache.h +0 -1
@@ -1669,7 +1669,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)extern int is_pack_valid (struct packed_git *);extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *); -extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep);extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *);
packfile.c +25 -0
@@ -884,3 +884,28 @@ void reprepare_packed_git(void) prepare_packed_git_run_once =0 ;prepare_packed_git (); } + +unsigned long unpack_object_header_buffer (const unsigned char *buf, +unsigned long len,enum object_type *type,unsigned long *sizep) +{ +unsigned shift; +unsigned long size, c; +unsigned long used =0 ; + + c = buf[used++]; + *type = (c >>4 ) &7 ; + size = c &15 ; + shift =4 ; +while (c &0x80 ) { +if (len <= used ||bitsizeof (long) <= shift) { +error ("bad object header" ); + size = used =0 ; +break ; + } + c = buf[used++]; + size += (c &0x7f ) << shift; + shift +=7 ; + } + *sizep = size; +return used; +}
packfile.h +2 -0
@@ -62,6 +62,8 @@ extern void close_all_packs(void);extern void unuse_pack (struct pack_window **);extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep); +extern void release_pack_memory (size_t );extern int open_packed_git (struct packed_git *p);
sha1_file.c +0 -25
@@ -914,31 +914,6 @@ void *map_sha1_file(const unsigned char *sha1, unsigned long *size)return map_sha1_file_1 (NULL , sha1, size); } -unsigned long unpack_object_header_buffer (const unsigned char *buf, -unsigned long len,enum object_type *type,unsigned long *sizep) -{ -unsigned shift; -unsigned long size, c; -unsigned long used =0 ; - - c = buf[used++]; - *type = (c >>4 ) &7 ; - size = c &15 ; - shift =4 ; -while (c &0x80 ) { -if (len <= used ||bitsizeof (long) <= shift) { -error ("bad object header" ); - size = used =0 ; -break ; - } - c = buf[used++]; - size += (c &0x7f ) << shift; - shift +=7 ; - } - *sizep = size; -return used; -} -static int unpack_sha1_short_header (git_zstream *stream,unsigned char *map,unsigned long mapsize,void *buffer,unsigned long bufsiz)
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 13/23 pack: move get_size_from_delta()
In reply to Jonathan Tan--- cache.h | 1 - packfile.c | 40 ++++++++++++++++++++++++++++++++++++++++ packfile.h | 1 + sha1_file.c | 39 --------------------------------------- 4 files changed, 41 insertions(+), 40 deletions(-)
cache.h +0 -1
@@ -1669,7 +1669,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)extern int is_pack_valid (struct packed_git *);extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *); -extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *);/*
packfile.c +40 -0
@@ -4,6 +4,7 @@#include "dir.h" #include "mergesort.h" #include "packfile.h" +#include "delta.h" char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1,
@@ -909,3 +910,42 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf, *sizep = size;return used; } + +unsigned long get_size_from_delta (struct packed_git *p, +struct pack_window **w_curs, +off_t curpos) +{ +const unsigned char *data; +unsigned char delta_head[20 ], *in; +git_zstream stream; +int st; + +memset (&stream,0 ,sizeof (stream)); + stream.next_out = delta_head; + stream.avail_out =sizeof (delta_head); + +git_inflate_init (&stream); +do { + in =use_pack (p, w_curs, curpos, &stream.avail_in); + stream.next_in = in; + st =git_inflate (&stream,Z_FINISH ); + curpos += stream.next_in - in; + }while ((st ==Z_OK || st ==Z_BUF_ERROR ) && + stream.total_out <sizeof (delta_head)); +git_inflate_end (&stream); +if ((st !=Z_STREAM_END ) && stream.total_out !=sizeof (delta_head)) { +error ("delta data unpack-initial failed" ); +return 0 ; + } + +/* Examine the initial part of the delta to figure out +* the result size. +*/ + data = delta_head; + +/* ignore base size */ +get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); + +/* Read the result size */ +return get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); +}
packfile.h +1 -0
@@ -63,6 +63,7 @@ extern void unuse_pack(struct pack_window **);extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local);extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep); +extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );extern void release_pack_memory (size_t );
sha1_file.c +0 -39
@@ -1100,45 +1100,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -unsigned long get_size_from_delta (struct packed_git *p, -struct pack_window **w_curs, -off_t curpos) -{ -const unsigned char *data; -unsigned char delta_head[20 ], *in; -git_zstream stream; -int st; - -memset (&stream,0 ,sizeof (stream)); - stream.next_out = delta_head; - stream.avail_out =sizeof (delta_head); - -git_inflate_init (&stream); -do { - in =use_pack (p, w_curs, curpos, &stream.avail_in); - stream.next_in = in; - st =git_inflate (&stream,Z_FINISH ); - curpos += stream.next_in - in; - }while ((st ==Z_OK || st ==Z_BUF_ERROR ) && - stream.total_out <sizeof (delta_head)); -git_inflate_end (&stream); -if ((st !=Z_STREAM_END ) && stream.total_out !=sizeof (delta_head)) { -error ("delta data unpack-initial failed" ); -return 0 ; - } - -/* Examine the initial part of the delta to figure out -* the result size. -*/ - data = delta_head; - -/* ignore base size */ -get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); - -/* Read the result size */ -return get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); -} - static off_t get_delta_base(struct packed_git *p, struct pack_window **w_curs, off_t *curpos,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 14/23 pack: move unpack_object_header()
In reply to Jonathan Tan--- cache.h | 1 - packfile.c | 26 ++++++++++++++++++++++++++ packfile.h | 1 + sha1_file.c | 26 -------------------------- 4 files changed, 27 insertions(+), 27 deletions(-)
cache.h +0 -1
@@ -1669,7 +1669,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)extern int is_pack_valid (struct packed_git *);extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *); -extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *);/* * Iterate over the files in the loose-object parts of the object
packfile.c +26 -0
@@ -949,3 +949,29 @@ unsigned long get_size_from_delta(struct packed_git *p,/* Read the result size */ return get_delta_hdr_size (&data, delta_head+sizeof (delta_head)); } + +int unpack_object_header (struct packed_git *p, +struct pack_window **w_curs, +off_t *curpos, +unsigned long *sizep) +{ +unsigned char *base; +size_t left; +size_t used; +enum object_type type; + +/* use_pack() assures us we have [base, base + 20) available +* as a range that we can look at. (Its actually the hash +* size that is assured.) With our object header encoding +* the maximum deflated object size is 2^137, which is just +* insane, so we know won't exceed what we have been given. +*/ + base =use_pack (p, w_curs, *curpos, &left); + used =unpack_object_header_buffer (base, left, &type, sizep); +if (!used) { + type =OBJ_BAD ; + }else + *curpos += used; + +return type; +}
packfile.h +1 -0
@@ -64,6 +64,7 @@ extern struct packed_git *add_packed_git(const char *path, size_t path_len, intextern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep);extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t ); +extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *);extern void release_pack_memory (size_t );
sha1_file.c +0 -26
@@ -1171,32 +1171,6 @@ static const unsigned char *get_delta_base_sha1(struct packed_git *p,return NULL ; } -int unpack_object_header (struct packed_git *p, -struct pack_window **w_curs, -off_t *curpos, -unsigned long *sizep) -{ -unsigned char *base; -size_t left; -size_t used; -enum object_type type; - -/* use_pack() assures us we have [base, base + 20) available -* as a range that we can look at. (Its actually the hash -* size that is assured.) With our object header encoding -* the maximum deflated object size is 2^137, which is just -* insane, so we know won't exceed what we have been given. -*/ - base =use_pack (p, w_curs, *curpos, &left); - used =unpack_object_header_buffer (base, left, &type, sizep); -if (!used) { - type =OBJ_BAD ; - }else - *curpos += used; - -return type; -} -static int retry_bad_packed_offset (struct packed_git *p,off_t obj_offset) {int type;
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 15/23 pack: move clear_delta_base_cache(), packed_object_info(), unpack_entry()
In reply to Jonathan TanBoth sha1_file.c and packfile.c now need read_object(), so a copy of read_object() was created in packfile.c.
This patch makes both mark_bad_packed_object() and has_packed_and_bad() global. Unlike most of the other patches in this series, these 2 functions need to remain global.
--- cache.h | 7 - packfile.c | 661 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ packfile.h | 10 + sha1_file.c | 677 ++---------------------------------------------------------- 4 files changed, 685 insertions(+), 670 deletions(-)
cache.h +0 -7
@@ -1194,9 +1194,6 @@ extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);extern int unpack_sha1_header (git_zstream *stream,unsigned char *map,unsigned long mapsize,void *buffer,unsigned long bufsiz);extern int parse_sha1_header (const char *hdr,unsigned long *sizep); -/* global flag to enable extra checks when accessing packed objects */ -extern int do_check_packed_object_crc; -extern int check_sha1_signature (const unsigned char *sha1,void *buf,unsigned long size,const char *type);extern int finalize_object_file (const char *tmpfile,const char *filename);
@@ -1629,8 +1626,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -extern void clear_delta_base_cache (void ); -/* * Make sure that a pointer access into an mmap'd index file is within bounds, * and can provide at least 8 bytes of data.
@@ -1668,7 +1663,6 @@ extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);extern off_t find_pack_entry_one (const unsigned char *sha1,struct packed_git *);extern int is_pack_valid (struct packed_git *); -extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *);/* * Iterate over the files in the loose-object parts of the object
@@ -1779,7 +1773,6 @@ struct object_info {/* Do not retry packed storage after checking packed and loose storage */ #define OBJECT_INFO_QUICK 8extern int sha1_object_info_extended (const unsigned char *,struct object_info *,unsigned flags); -extern int packed_object_info (struct packed_git *pack,off_t offset,struct object_info *);/* Dumb servers support */ extern int update_server_info (int );
packfile.c +661 -0
670 lines not on this page · raw message
packfile.h +10 -0
@@ -60,8 +60,10 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_textern void close_pack_windows (struct packed_git *);extern void close_all_packs (void );extern void unuse_pack (struct pack_window **); +extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *);extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep);extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );extern int unpack_object_header (struct packed_git *,struct pack_window **,off_t *,unsigned long *);
@@ -70,4 +72,12 @@ extern void release_pack_memory(size_t);extern int open_packed_git (struct packed_git *p); +/* global flag to enable extra checks when accessing packed objects */ +extern int do_check_packed_object_crc; + +extern int packed_object_info (struct packed_git *pack,off_t offset,struct object_info *); + +extern void mark_bad_packed_object (struct packed_git *p,const unsigned char *sha1); +extern const struct packed_git *has_packed_and_bad (const unsigned char *sha1); +#endif
sha1_file.c +14 -663
701 lines not on this page · raw message
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 16/23 pack: move nth_packed_object_{sha1,oid}
In reply to Jonathan Tan--- cache.h | 14 -------------- packfile.c | 31 +++++++++++++++++++++++++++++++ packfile.h | 16 +++++++++++++++- sha1_file.c | 31 ------------------------------- 4 files changed, 46 insertions(+), 46 deletions(-)
cache.h +0 -14
@@ -1636,20 +1636,6 @@ extern int odb_pack_keep(const char *name);*/ extern void check_pack_index_ptr (const struct packed_git *p,const void *ptr); -/* -* Return the SHA-1 of the nth object within the specified packfile. -* Open the index if it is not already open. The return value points -* at the SHA-1 within the mmapped index. Return NULL if there is an -* error. -*/ -extern const unsigned char *nth_packed_object_sha1 (struct packed_git *,uint32_t n); -/* -* Like nth_packed_object_sha1, but write the data into the object specified by -* the the first argument. Returns the first argument on success, and NULL on -* error. -*/ -extern const struct object_id *nth_packed_object_oid (struct object_id *,struct packed_git *,uint32_t n); -/* * Return the offset of the nth object within the specified packfile. * The index must already be opened.
packfile.c +31 -0
@@ -1636,3 +1636,34 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,return data; } + +const unsigned char *nth_packed_object_sha1 (struct packed_git *p, +uint32_t n) +{ +const unsigned char *index = p->index_data; +if (!index) { +if (open_pack_index (p)) +return NULL ; + index = p->index_data; + } +if (n >= p->num_objects) +return NULL ; + index +=4 *256 ; +if (p->index_version ==1 ) { +return index +24 * n +4 ; + }else { + index +=8 ; +return index +20 * n; + } +} + +const struct object_id *nth_packed_object_oid (struct object_id *oid, +struct packed_git *p, +uint32_t n) +{ +const unsigned char *hash =nth_packed_object_sha1 (p, n); +if (!hash) +return NULL ; +hashcpy (oid->hash, hash); +return oid; +}
packfile.h +15 -1
@@ -63,6 +63,21 @@ extern void unuse_pack(struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +/* +* Return the SHA-1 of the nth object within the specified packfile. +* Open the index if it is not already open. The return value points +* at the SHA-1 within the mmapped index. Return NULL if there is an +* error. +*/ +extern const unsigned char *nth_packed_object_sha1 (struct packed_git *,uint32_t n); +/* +* Like nth_packed_object_sha1, but write the data into the object specified by +* the the first argument. Returns the first argument on success, and NULL on +* error. +*/ +extern const struct object_id *nth_packed_object_oid (struct object_id *,struct packed_git *,uint32_t n); + +extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *);extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep);extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );
@@ -79,5 +94,4 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct objeextern void mark_bad_packed_object (struct packed_git *p,const unsigned char *sha1);extern const struct packed_git *has_packed_and_bad (const unsigned char *sha1); -#endif
sha1_file.c +0 -31
@@ -1074,37 +1074,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -const unsigned char *nth_packed_object_sha1 (struct packed_git *p, -uint32_t n) -{ -const unsigned char *index = p->index_data; -if (!index) { -if (open_pack_index (p)) -return NULL ; - index = p->index_data; - } -if (n >= p->num_objects) -return NULL ; - index +=4 *256 ; -if (p->index_version ==1 ) { -return index +24 * n +4 ; - }else { - index +=8 ; -return index +20 * n; - } -} - -const struct object_id *nth_packed_object_oid (struct object_id *oid, -struct packed_git *p, -uint32_t n) -{ -const unsigned char *hash =nth_packed_object_sha1 (p, n); -if (!hash) -return NULL ; -hashcpy (oid->hash, hash); -return oid; -} -void check_pack_index_ptr (const struct packed_git *p,const void *vptr) {const unsigned char *ptr = vptr;
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 17/23 pack: move check_pack_index_ptr(), nth_packed_object_offset()
In reply to Jonathan Tan--- cache.h | 16 ---------------- packfile.c | 33 +++++++++++++++++++++++++++++++++ packfile.h | 16 ++++++++++++++++ sha1_file.c | 33 --------------------------------- 4 files changed, 49 insertions(+), 49 deletions(-)
cache.h +0 -16
@@ -1626,22 +1626,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* Make sure that a pointer access into an mmap'd index file is within bounds, -* and can provide at least 8 bytes of data. -* -* Note that this is only necessary for variable-length segments of the file -* (like the 64-bit extended offset table), as we compare the size to the -* fixed-length parts when we open the file. -*/ -extern void check_pack_index_ptr (const struct packed_git *p,const void *ptr); - -/* -* Return the offset of the nth object within the specified packfile. -* The index must already be opened. -*/ -extern off_t nth_packed_object_offset (const struct packed_git *,uint32_t n); -/* * If the object named sha1 is present in the specified packfile, * return its offset within the packfile; otherwise, return 0.
packfile.c +33 -0
@@ -1667,3 +1667,36 @@ const struct object_id *nth_packed_object_oid(struct object_id *oid,hashcpy (oid->hash, hash);return oid; } + +void check_pack_index_ptr (const struct packed_git *p,const void *vptr) +{ +const unsigned char *ptr = vptr; +const unsigned char *start = p->index_data; +const unsigned char *end = start + p->index_size; +if (ptr < start) +die (_ ("offset before start of pack index for %s (corrupt index?)" ), + p->pack_name); +/* No need to check for underflow; .idx files must be at least 8 bytes */ +if (ptr >= end -8 ) +die (_ ("offset beyond end of pack index for %s (truncated index?)" ), + p->pack_name); +} + +off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) +{ +const unsigned char *index = p->index_data; + index +=4 *256 ; +if (p->index_version ==1 ) { +return ntohl (*((uint32_t *)(index +24 * n))); + }else { +uint32_t off; + index +=8 + p->num_objects * (20 +4 ); + off =ntohl (*((uint32_t *)(index +4 * n))); +if (!(off &0x80000000 )) +return off; + index += p->num_objects *4 + (off &0x7fffffff ) *8 ; +check_pack_index_ptr (p, index); +return (((uint64_t )ntohl (*((uint32_t *)(index +0 )))) <<32 ) | +ntohl (*((uint32_t *)(index +4 ))); + } +}
packfile.h +16 -0
@@ -63,6 +63,16 @@ extern void unuse_pack(struct pack_window **);extern void clear_delta_base_cache (void );extern struct packed_git *add_packed_git (const char *path,size_t path_len,int local); +/* +* Make sure that a pointer access into an mmap'd index file is within bounds, +* and can provide at least 8 bytes of data. +* +* Note that this is only necessary for variable-length segments of the file +* (like the 64-bit extended offset table), as we compare the size to the +* fixed-length parts when we open the file. +*/ +extern void check_pack_index_ptr (const struct packed_git *p,const void *ptr); +/* * Return the SHA-1 of the nth object within the specified packfile. * Open the index if it is not already open. The return value points
@@ -77,6 +87,11 @@ extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t*/ extern const struct object_id *nth_packed_object_oid (struct object_id *,struct packed_git *,uint32_t n); +/* +* Return the offset of the nth object within the specified packfile. +* The index must already be opened. +*/ +extern off_t nth_packed_object_offset (const struct packed_git *,uint32_t n);extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *);extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep);
@@ -94,4 +109,5 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct objeextern void mark_bad_packed_object (struct packed_git *p,const unsigned char *sha1);extern const struct packed_git *has_packed_and_bad (const unsigned char *sha1); +#endif
sha1_file.c +0 -33
@@ -1074,39 +1074,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)return parse_sha1_header_extended (hdr, &oi,0 ); } -void check_pack_index_ptr (const struct packed_git *p,const void *vptr) -{ -const unsigned char *ptr = vptr; -const unsigned char *start = p->index_data; -const unsigned char *end = start + p->index_size; -if (ptr < start) -die (_ ("offset before start of pack index for %s (corrupt index?)" ), - p->pack_name); -/* No need to check for underflow; .idx files must be at least 8 bytes */ -if (ptr >= end -8 ) -die (_ ("offset beyond end of pack index for %s (truncated index?)" ), - p->pack_name); -} - -off_t nth_packed_object_offset (const struct packed_git *p,uint32_t n) -{ -const unsigned char *index = p->index_data; - index +=4 *256 ; -if (p->index_version ==1 ) { -return ntohl (*((uint32_t *)(index +24 * n))); - }else { -uint32_t off; - index +=8 + p->num_objects * (20 +4 ); - off =ntohl (*((uint32_t *)(index +4 * n))); -if (!(off &0x80000000 )) -return off; - index += p->num_objects *4 + (off &0x7fffffff ) *8 ; -check_pack_index_ptr (p, index); -return (((uint64_t )ntohl (*((uint32_t *)(index +0 )))) <<32 ) | -ntohl (*((uint32_t *)(index +4 ))); - } -} -off_t find_pack_entry_one (const unsigned char *sha1,struct packed_git *p) {
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 18/23 pack: move find_pack_entry_one(), is_pack_valid()
In reply to Jonathan Tan--- cache.h | 8 ------- packfile.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++- packfile.h | 9 ++++++-- sha1_file.c | 73 ---------------------------------------------------------- 4 files changed, 82 insertions(+), 84 deletions(-)
cache.h +0 -8
@@ -1626,14 +1626,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);*/ extern int odb_pack_keep (const char *name); -/* -* If the object named sha1 is present in the specified packfile, -* return its offset within the packfile; otherwise, return 0. -*/ -extern off_t find_pack_entry_one (const unsigned char *sha1,struct packed_git *); - -extern int is_pack_valid (struct packed_git *); -/* * Iterate over the files in the loose-object parts of the object * directory "path", triggering the following callbacks:
packfile.c +75 -1
91 lines not on this page · raw message
packfile.h +7 -2
@@ -93,6 +93,13 @@ extern const struct object_id *nth_packed_object_oid(struct object_id *, struct*/ extern off_t nth_packed_object_offset (const struct packed_git *,uint32_t n); +/* +* If the object named sha1 is present in the specified packfile, +* return its offset within the packfile; otherwise, return 0. +*/ +extern off_t find_pack_entry_one (const unsigned char *sha1,struct packed_git *); + +extern int is_pack_valid (struct packed_git *);extern void *unpack_entry (struct packed_git *,off_t ,enum object_type *,unsigned long *);extern unsigned long unpack_object_header_buffer (const unsigned char *buf,unsigned long len,enum object_type *type,unsigned long *sizep);extern unsigned long get_size_from_delta (struct packed_git *,struct pack_window **,off_t );
@@ -100,8 +107,6 @@ extern int unpack_object_header(struct packed_git *, struct pack_window **, off_extern void release_pack_memory (size_t ); -extern int open_packed_git (struct packed_git *p); -/* global flag to enable extra checks when accessing packed objects */ extern int do_check_packed_object_crc;
sha1_file.c +0 -73
79 lines not on this page · raw message
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 19/23 pack: move find_sha1_pack()
In reply to Jonathan Tan--- cache.h | 3 --- http-push.c | 1 + http-walker.c | 1 + packfile.c | 13 +++++++++++++ packfile.h | 3 +++ sha1_file.c | 13 ------------- 6 files changed, 18 insertions(+), 16 deletions(-)
cache.h +0 -3
@@ -1608,9 +1608,6 @@ struct pack_entry {struct packed_git *p; }; -extern struct packed_git *find_sha1_pack (const unsigned char *sha1, -struct packed_git *packs); -/* * Create a temporary file rooted in the object database directory, or * die on failure. The filename is taken from "pattern", which should have the
http-push.c +1 -0
@@ -11,6 +11,7 @@#include "list-objects.h" #include "sigchain.h" #include "argv-array.h" +#include "packfile.h" #ifdef EXPAT_NEEDS_XMLPARSE_H #include <xmlparse.h>
http-walker.c +1 -0
@@ -4,6 +4,7 @@#include "http.h" #include "list.h" #include "transport.h" +#include "packfile.h" struct alt_base {char *base;
packfile.c +13 -0
@@ -1774,3 +1774,16 @@ int is_pack_valid(struct packed_git *p)/* Force the pack to open to prove its valid. */ return !open_packed_git (p); } + +struct packed_git *find_sha1_pack (const unsigned char *sha1, +struct packed_git *packs) +{ +struct packed_git *p; + +for (p = packs; p; p = p->next) { +if (find_pack_entry_one (sha1, p)) +return p; + } +return NULL ; + +}
packfile.h +3 -0
@@ -42,6 +42,9 @@ extern void install_packed_git(struct packed_git *pack);*/ unsigned long approximate_object_count (void ); +extern struct packed_git *find_sha1_pack (const unsigned char *sha1, +struct packed_git *packs); +extern void pack_report (void );/*
sha1_file.c +0 -13
@@ -1127,19 +1127,6 @@ static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)return 0 ; } -struct packed_git *find_sha1_pack (const unsigned char *sha1, -struct packed_git *packs) -{ -struct packed_git *p; - -for (p = packs; p; p = p->next) { -if (find_pack_entry_one (sha1, p)) -return p; - } -return NULL ; - -} -static int sha1_loose_object_info (const unsigned char *sha1,struct object_info *oi,int flags)
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 20/23 pack: move find_pack_entry() and make it global
In reply to Jonathan TanThis function needs to be global as it is used by sha1_file.c and will be used by packfile.c.
--- packfile.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++ packfile.h | 2 ++ sha1_file.c | 53 ----------------------------------------------------- 3 files changed, 55 insertions(+), 53 deletions(-)
packfile.c +53 -0
@@ -1787,3 +1787,56 @@ struct packed_git *find_sha1_pack(const unsigned char *sha1,return NULL ; } + +static int fill_pack_entry (const unsigned char *sha1, +struct pack_entry *e, +struct packed_git *p) +{ +off_t offset; + +if (p->num_bad_objects) { +unsigned i; +for (i =0 ; i < p->num_bad_objects; i++) +if (!hashcmp (sha1, p->bad_object_sha1 +20 * i)) +return 0 ; + } + + offset =find_pack_entry_one (sha1, p); +if (!offset) +return 0 ; + +/* +* We are about to tell the caller where they can locate the +* requested object. We better make sure the packfile is +* still here and can be accessed before supplying that +* answer, as it may have been deleted since the index was +* loaded! +*/ +if (!is_pack_valid (p)) +return 0 ; + e->offset = offset; + e->p = p; +hashcpy (e->sha1, sha1); +return 1 ; +} + +/* +* Iff a pack file contains the object named by sha1, return true and +* store its location to e. +*/ +int find_pack_entry (const unsigned char *sha1,struct pack_entry *e) +{ +struct mru_entry *p; + +prepare_packed_git (); +if (!packed_git) +return 0 ; + +for (p = packed_git_mru->head; p; p = p->next) { +if (fill_pack_entry (sha1, e, p->item)) { +mru_mark (packed_git_mru, p); +return 1 ; + } + } +return 0 ; +}
packfile.h +2 -0
@@ -118,4 +118,6 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct objeextern void mark_bad_packed_object (struct packed_git *p,const unsigned char *sha1);extern const struct packed_git *has_packed_and_bad (const unsigned char *sha1); +extern int find_pack_entry (const unsigned char *sha1,struct pack_entry *e); +#endif
sha1_file.c +0 -53
59 lines not on this page · raw message
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 21/23 pack: move has_sha1_pack()
In reply to Jonathan Tan--- builtin/prune-packed.c | 1 + cache.h | 2 -- diff.c | 1 + packfile.c | 6 ++++++ packfile.h | 2 ++ revision.c | 1 + sha1_file.c | 6 ------ 7 files changed, 11 insertions(+), 8 deletions(-)
builtin/prune-packed.c +1 -0
@@ -2,6 +2,7 @@#include "cache.h" #include "progress.h" #include "parse-options.h" +#include "packfile.h" static const char *const prune_packed_usage[] = {N_ ("git prune-packed [-n | --dry-run] [-q | --quiet]" ),
cache.h +0 -2
@@ -1198,8 +1198,6 @@ extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned lextern int finalize_object_file (const char *tmpfile,const char *filename); -extern int has_sha1_pack (const unsigned char *sha1); -/* * Open the loose object at path, check its sha1, and return the contents, * type, and size. If the object is a blob, then "contents" may return NULL,
diff.c +1 -0
@@ -20,6 +20,7 @@#include "string-list.h" #include "argv-array.h" #include "graph.h" +#include "packfile.h" #ifdef NO_FAST_WORKING_DIRECTORY #define FAST_WORKING_DIRECTORY 0
packfile.c +6 -0
@@ -1840,3 +1840,9 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e) }return 0 ; } + +int has_sha1_pack (const unsigned char *sha1) +{ +struct pack_entry e; +return find_pack_entry (sha1, &e); +}
packfile.h +2 -0
@@ -120,4 +120,6 @@ extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);extern int find_pack_entry (const unsigned char *sha1,struct pack_entry *e); +extern int has_sha1_pack (const unsigned char *sha1); +#endif
revision.c +1 -0
@@ -19,6 +19,7 @@#include "dir.h" #include "cache-tree.h" #include "bisect.h" +#include "packfile.h" volatile show_early_output_fn_t show_early_output;
sha1_file.c +0 -6
@@ -1630,12 +1630,6 @@ int has_pack_index(const unsigned char *sha1)return 1 ; } -int has_sha1_pack (const unsigned char *sha1) -{ -struct pack_entry e; -return find_pack_entry (sha1, &e); -} -int has_sha1_file_with_flags (const unsigned char *sha1,int flags) {if (!startup_info->have_repository)
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 22/23 pack: move has_pack_index()
In reply to Jonathan Tan--- cache.h | 2 -- packfile.c | 8 ++++++++ packfile.h | 2 ++ sha1_file.c | 8 -------- 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -2
@@ -1233,8 +1233,6 @@ extern int has_object_file_with_flags(const struct object_id *oid, int flags);*/ extern int has_loose_object_nonlocal (const unsigned char *sha1); -extern int has_pack_index (const unsigned char *sha1); -extern void assert_sha1_type (const unsigned char *sha1,enum object_type expect);/* Helper to check and "touch" a file */
packfile.c +8 -0
@@ -1846,3 +1846,11 @@ int has_sha1_pack(const unsigned char *sha1)struct pack_entry e;return find_pack_entry (sha1, &e); } + +int has_pack_index (const unsigned char *sha1) +{ +struct stat st; +if (stat (sha1_pack_index_name (sha1), &st)) +return 0 ; +return 1 ; +}
packfile.h +2 -0
@@ -122,4 +122,6 @@ extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);extern int has_sha1_pack (const unsigned char *sha1); +extern int has_pack_index (const unsigned char *sha1); +#endif
sha1_file.c +0 -8
@@ -1622,14 +1622,6 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)return ret; } -int has_pack_index (const unsigned char *sha1) -{ -struct stat st; -if (stat (sha1_pack_index_name (sha1), &st)) -return 0 ; -return 1 ; -} -int has_sha1_file_with_flags (const unsigned char *sha1,int flags) {if (!startup_info->have_repository)
Signature · 1 line
2.14.1.480.gb18f417b89-goog
PATCH v3 23/23 pack: move for_each_packed_object()
In reply to Jonathan Tan--- builtin/cat-file.c | 1 + cache.h | 7 +------ packfile.c | 40 ++++++++++++++++++++++++++++++++++++++++ packfile.h | 11 +++++++++++ reachable.c | 1 + sha1_file.c | 40 ---------------------------------------- 6 files changed, 54 insertions(+), 46 deletions(-)
builtin/cat-file.c +1 -0
@@ -12,6 +12,7 @@#include "streaming.h" #include "tree-walk.h" #include "sha1-array.h" +#include "packfile.h" struct batch_options {int enabled;
cache.h +1 -6
@@ -1668,17 +1668,12 @@ int for_each_loose_file_in_objdir_buf(struct strbuf *path, void *data);/* -* Iterate over loose and packed objects in both the local +* Iterate over loose objects in both the local * repository and any alternates repositories (unless the * LOCAL_ONLY flag is set). */ #define FOR_EACH_OBJECT_LOCAL_ONLY 0x1 -typedef int each_packed_object_fn (const struct object_id *oid, -struct packed_git *pack, -uint32_t pos, -void *data);extern int for_each_loose_object (each_loose_object_fn ,void *,unsigned flags); -extern int for_each_packed_object (each_packed_object_fn ,void *,unsigned flags);struct object_info {/* Request */
packfile.c +40 -0
@@ -1854,3 +1854,43 @@ int has_pack_index(const unsigned char *sha1)return 0 ;return 1 ; } + +static int for_each_object_in_pack (struct packed_git *p,each_packed_object_fn cb,void *data) +{ +uint32_t i; +int r =0 ; + +for (i =0 ; i < p->num_objects; i++) { +struct object_id oid; + +if (!nth_packed_object_oid (&oid, p, i)) +return error ("unable to get sha1 of object %u in %s" , + i, p->pack_name); + + r =cb (&oid, p, i, data); +if (r) +break ; + } +return r; +} + +int for_each_packed_object (each_packed_object_fn cb,void *data,unsigned flags) +{ +struct packed_git *p; +int r =0 ; +int pack_errors =0 ; + +prepare_packed_git (); +for (p = packed_git; p; p = p->next) { +if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local) +continue ; +if (open_pack_index (p)) { + pack_errors =1 ; +continue ; + } + r =for_each_object_in_pack (p, cb, data); +if (r) +break ; + } +return r ? r : pack_errors; +}
packfile.h +11 -0
@@ -124,4 +124,15 @@ extern int has_sha1_pack(const unsigned char *sha1);extern int has_pack_index (const unsigned char *sha1); +/* +* Iterate over packed objects in both the local +* repository and any alternates repositories (unless the +* FOR_EACH_OBJECT_LOCAL_ONLY flag, defined in cache.h, is set). +*/ +typedef int each_packed_object_fn (const struct object_id *oid, +struct packed_git *pack, +uint32_t pos, +void *data); +extern int for_each_packed_object (each_packed_object_fn ,void *,unsigned flags); +#endif
reachable.c +1 -0
@@ -9,6 +9,7 @@#include "cache-tree.h" #include "progress.h" #include "list-objects.h" +#include "packfile.h" struct connectivity_progress {struct progress *progress;
sha1_file.c +0 -40
@@ -2014,46 +2014,6 @@ int for_each_loose_object(each_loose_object_fn cb, void *data, unsigned flags)return foreach_alt_odb (loose_from_alt_odb, &alt); } -static int for_each_object_in_pack (struct packed_git *p,each_packed_object_fn cb,void *data) -{ -uint32_t i; -int r =0 ; - -for (i =0 ; i < p->num_objects; i++) { -struct object_id oid; - -if (!nth_packed_object_oid (&oid, p, i)) -return error ("unable to get sha1 of object %u in %s" , - i, p->pack_name); - - r =cb (&oid, p, i, data); -if (r) -break ; - } -return r; -} - -int for_each_packed_object (each_packed_object_fn cb,void *data,unsigned flags) -{ -struct packed_git *p; -int r =0 ; -int pack_errors =0 ; - -prepare_packed_git (); -for (p = packed_git; p; p = p->next) { -if ((flags &FOR_EACH_OBJECT_LOCAL_ONLY ) && !p->pack_local) -continue ; -if (open_pack_index (p)) { - pack_errors =1 ; -continue ; - } - r =for_each_object_in_pack (p, cb, data); -if (r) -break ; - } -return r ? r : pack_errors; -} - static int check_stream_sha1(git_zstream *stream, const char *hdr, unsigned long size,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
Re: [PATCH v3 00/23] Move exported packfile funcs to its own file
In reply to Jonathan TanQuoted 26 lines · Jonathan Tan
Quoted 8 lines
You'd need to double check, but I think the topics that cause trouble are rs/find-apck-entry-bisection and jk/drop-sha1-entry-pos; you can start from v2.14.1 and merge these topics on top and then build your change on top. That would allow you to start cooking before both of them graduate to 'master', as I expect they are both quick-to-next material. There might be other topics that interfere with what you are doing, but you can easily find out what they are if you do a trial merge to 'next' and 'pu' yourself.
OK - in addition to the 2 you mentioned, I have found some others (likely added after you wrote that). The complete list is:
- rs/find-pack-entry-bisection
- jk/drop-sha1-entry-pos
- jt/sha1-file-cleanup (formerly part of this set)
- mk/use-size-t-in-zlib
- rs/unpack-entry-leakfixI have merged all of these and rebased my patches on top.
Other changes:
- Usedpackfile.hinstead ofpack.h(following most people's preference)
- Ensured that I added functions topackfile.hretaining the order they were originally in, so that if you run "git diff <base> --color-moved --patience", there are much fewer zebra stripes
Quoted 3 lines
The merge base commit can be accessed online [1], if you need it.
[1] https://github.com/jonathantanmy/git/commits/packmigrate
Thanks.
I have to say that this was a painful topic to integrate.
As you may know, the mk/use-size-t-in-zlib topic is being retracted and getting rerolled as a larger size_t series, most of which still needs help in reviewing.
The jt/sha1-file-cleanup topic is the only one among the other four that are still not in 'next', and I think that topic, as well as the other three, are all good and serve as a good base to build on top. So I first rebuilt your patches on top of these four topics. This took some time but it wasn't all that painful.
The result cleanly merged to 'pu', I think, but it resulted in a rather noisy conflict when I attempted to merge it to 'next'. I want to see both of a merge to 'next' and a merge to 'pu' to be reasonably clean for any topic to be viable [*1*]. Otherwise, "initially queue in 'pu', then cook in 'next', and eventually graduate to 'master'" workflow would not work well.
Anyway, I _think_ I finally got the conflict resolutions right for merges of the topic to 'next', 'jch' and 'pu', so I will push the result of merging to 'pu' out. This unfortunately makes Martin's ongoing size_t topic unmergeable to any of the integration branches as-is, but let's make sure that topic is reviewed properly first (I haven't seen people comment much on the individual patches other than just selected few).
[Footnote]
*1* There is an intermediate point between 'master' and 'pu' called
'jch', and I try to make sure any new topic to merge cleanly to that branch, too, when accepting it. That is the branch I use for everyday work as an early guinea-pig.
Re: [PATCH v3 00/23] Move exported packfile funcs to its own file
Quoted 11 lines · Junio C Hamano
I have to say that this was a painful topic to integrate.
As you may know, the mk/use-size-t-in-zlib topic is being retracted and getting rerolled as a larger
size_tseries, most of which still needs help in reviewing.The jt/sha1-file-cleanup topic is the only one among the other four that are still not in 'next', and I think that topic, as well as the other three, are all good and serve as a good base to build on top. So I first rebuilt your patches on top of these four topics. This took some time but it wasn't all that painful.
... but it turns out that I screwed it up in at least one place, making Linux32 build fail (Thanks Lars and folks who pushed hard to arrange Travis to build all my pushes to 'pu'). I'm pushing out my second attempt. Let's see how it goes.
A change like this that only moves code around without changing anything is painful to everybody to keep around, as nobody can safely touch the affected code while it is in flight. On the other hand, as long as it is only moving code around, such a change is reasonably safe, and it is relatively easy to ensure that there is no change other than code movement is involved. So let's
(1) make sure that the topics this depends on are sound by re-reading them once again, and merge them quickly down to 'master';
(2) merge this topic to 'next', optionally after rebasing it on 'master', after (1) happens; and
(3) quickly merge it to 'master', to get it over with.
In the meantime we'd need to refrain from taking code that touch things that are moved by this series.
I plan to be offline for a week or so near the end of this month, so I am hoping that we can do all of the above before that. That may make us break our usual "tip of the 'master' is more stable and robust than any released version" promise by potentially leaving it broken for a while, but nobody can build on top of a fluid codebase that is in the process of moving things around in a big way, so it might not be such a bad idea to make it coincide with the period when the tree must become quiescent due to my being offline. We'll see.
Thanks.
Re: [PATCH v3 00/23] Move exported packfile funcs to its own file
Quoted 14 lines · On Sat, 19 Aug 2017 23:40:33 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 11 lines · Junio C Hamano
I have to say that this was a painful topic to integrate.
As you may know, the mk/use-size-t-in-zlib topic is being retracted and getting rerolled as a larger
size_tseries, most of which still needs help in reviewing.The jt/sha1-file-cleanup topic is the only one among the other four that are still not in 'next', and I think that topic, as well as the other three, are all good and serve as a good base to build on top. So I first rebuilt your patches on top of these four topics. This took some time but it wasn't all that painful.
Quoted 4 lines
... but it turns out that I screwed it up in at least one place, making Linux32 build fail (Thanks Lars and folks who pushed hard to arrange Travis to build all my pushes to 'pu'). I'm pushing out my second attempt. Let's see how it goes.
Thanks.
Quoted 10 lines
A change like this that only moves code around without changing anything is painful to everybody to keep around, as nobody can safely touch the affected code while it is in flight. On the other hand, as long as it is only moving code around, such a change is reasonably safe, and it is relatively easy to ensure that there is no change other than code movement is involved. So let's
(1) make sure that the topics this depends on are sound by re-reading them once again, and merge them quickly down to 'master';
I took a look and they look sound.
- rs/find-pack-entry-bisection resolves an issue first introduced in commit 1f68855 ("[PATCH] Teach read_sha1_file() and friends about packed git object store.", 2005-06-27), which already had that issue.
- jk/drop-sha1-entry-pos is some code deletion.
- rs/unpack-entry-leakfix ensures that delta_stack is freed. This function does not (for example) expose the destination of delta_stack to its caller, so it is correct that delta_stack should be freed unless it points to the local buffer, just like in the success case.
- jt/sha1-file-cleanup (my patches) still looks OK to me.
In your latest "What's cooking" (Aug 2017, #04; Fri, 18), you mentioned that the first 3 will be merged to master, and the 4th will be merged to next.
I didn't look at mk/use-size-t-in-zlib, which (as you said) is still under review.
Quoted 19 lines
(2) merge this topic to 'next', optionally after rebasing it on 'master', after (1) happens; and
(3) quickly merge it to 'master', to get it over with.
In the meantime we'd need to refrain from taking code that touch things that are moved by this series.
I plan to be offline for a week or so near the end of this month, so I am hoping that we can do all of the above before that. That may make us break our usual "tip of the 'master' is more stable and robust than any released version" promise by potentially leaving it broken for a while, but nobody can build on top of a fluid codebase that is in the process of moving things around in a big way, so it might not be such a bad idea to make it coincide with the period when the tree must become quiescent due to my being offline. We'll see.
Thanks.
Re: [PATCH v3 00/23] Move exported packfile funcs to its own file
Quoted 17 lines · Jonathan Tan
Quoted 14 lines · On Sat, 19 Aug 2017 23:40:33 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 11 lines · Junio C Hamano
I have to say that this was a painful topic to integrate.
As you may know, the mk/use-size-t-in-zlib topic is being retracted and getting rerolled as a larger
size_tseries, most of which still needs help in reviewing.The jt/sha1-file-cleanup topic is the only one among the other four that are still not in 'next', and I think that topic, as well as the other three, are all good and serve as a good base to build on top. So I first rebuilt your patches on top of these four topics. This took some time but it wasn't all that painful.
Quoted 6 lines
Quoted 4 lines
... but it turns out that I screwed it up in at least one place, making Linux32 build fail (Thanks Lars and folks who pushed hard to arrange Travis to build all my pushes to 'pu'). I'm pushing out my second attempt. Let's see how it goes.
Thanks.
It seems like a later pushout late Sunday night that had the second attempt made it pass on Linux32 ;-) Whew.
Quoted 16 lines
Quoted 3 lines
(1) make sure that the topics this depends on are sound by re-reading them once again, and merge them quickly down to 'master';
I took a look and they look sound.
- rs/find-pack-entry-bisection resolves an issue first introduced in commit 1f68855 ("[PATCH] Teach
read_sha1_file()and friends about packed git object store.", 2005-06-27), which already had that issue.
- jk/drop-sha1-entry-pos is some code deletion.
- rs/unpack-entry-leakfix ensures thatdelta_stackis freed. This function does not (for example) expose the destination ofdelta_stackto its caller, so it is correct thatdelta_stackshould be freed unless it points to the local buffer, just like in the success case.
- jt/sha1-file-cleanup (my patches) still looks OK to me.
Quoted 3 lines
In your latest "What's cooking" (Aug 2017, #04; Fri, 18), you mentioned that the first 3 will be merged to master, and the 4th will be merged to next.
Yup, thanks for double checking. I'll be merging them down soon-ish.
Thanks.