Skip to contentSkip to conversation

An attempt to move packfile funcs to its own file

RFC PATCH Superseded Jonathan Tan 5 participants 88 messages Aug 8, 2017 – Aug 21, 2017 Kbuild

lore
Share & tools
lore
#0 lore raw

cover An attempt to move packfile funcs to its own file

While investigating annotating packfiles and loose objects to support connectivity checks in partial clones [1], I decided to make the effort to separate packfile-related code from sha1_file.c into its own file, to make it easier to both code such changes and review them. Here is the beginning of those efforts.

Is this something worth doing, and if yes, is this in the right direction?

[1] https://public-inbox.org/git/[email protected]/

Jonathan Tan (10):
pack: move pack name-related functions
pack: move static state variables
pack: move pack_report()
pack: move open_pack_index(), parse_pack_index()
pack: move release_pack_memory()
pack: move pack-closing functions
pack: move use_pack()
pack: move unuse_pack()
pack: move add_packed_git()
pack: move install_packed_git()

 Makefile                 |   1 +
 builtin/am.c             |   1 +
 builtin/clone.c          |   1 +
 builtin/count-objects.c  |   1 +
 builtin/fetch.c          |   1 +
 builtin/merge.c          |   1 +
 builtin/pack-redundant.c |   1 +
 cache.h                  |  45 ----
 connected.c              |   1 +
 git-compat-util.h        |   2 -
 pack.c                   | 669 +++++++++++++++++++++++++++++++++++++++++++++++
 pack.h                   |  51 ++++
 sha1_file.c              | 667 ----------------------------------------------
 sha1_name.c              |   1 +
 streaming.c              |   1 +
 15 files changed, 730 insertions(+), 714 deletions(-)
 create mode 100644 pack.c
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#1 lore raw

RFC PATCH 1/10 pack: move pack name-related functions

Currently, sha1_file.c and cache.h contain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.

Create a new file, pack.c, to hold all packfile-related functions currently in sha1_file.c, and designate pack.h to hold these packfile-related functions.

In this commit, the pack name-related functions are moved. Subsequent commits will move the other functions.

---
 Makefile                 |  1 +
 builtin/pack-redundant.c |  1 +
 cache.h                  | 23 -----------------------
 pack.c                   | 23 +++++++++++++++++++++++
 pack.h                   | 23 +++++++++++++++++++++++
 sha1_file.c              | 22 ----------------------
 6 files changed, 48 insertions(+), 45 deletions(-)
 create mode 100644 pack.c
Makefile +1 -0
index 461c845d3..a7b901a18 100644
@@ -816,6 +816,7 @@ LIB_OBJS += notes-merge.o
 LIB_OBJS += notes-utils.o
 LIB_OBJS += object.o
 LIB_OBJS += oidset.o
+LIB_OBJS += pack.o
 LIB_OBJS += pack-bitmap.o
 LIB_OBJS += pack-bitmap-write.o
 LIB_OBJS += pack-check.o
builtin/pack-redundant.c +1 -0
index cb1df1c76..df36d10e7 100644
@@ -7,6 +7,7 @@
 */
 
 #include "builtin.h"
+#include "pack.h"
 
 #define BLKSIZE 512
 
cache.h +0 -23
index 71fe09264..1f0f47819 100644
@@ -902,20 +902,6 @@ extern void check_repository_format(void);
  */
 extern const char *sha1_file_name(const unsigned char *sha1);
 
-/*
- * Return the name of the (local) packfile with the specified sha1 in
- * its name.  The return value is a pointer to memory that is
- * overwritten each time this function is called.
- */
-extern char *sha1_pack_name(const unsigned char *sha1);
-
-/*
- * Return the name of the (local) pack index file with the specified
- * sha1 in its name.  The return value is a pointer to memory that is
- * overwritten each time this function is called.
- */
-extern char *sha1_pack_index_name(const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
@@ -1648,15 +1634,6 @@ extern void pack_report(void);
  */
 extern int odb_mkstemp(struct strbuf *template, const char *pattern);
 
-/*
- * Generate the filename to be used for a pack file with checksum "sha1" and
- * extension "ext". The result is written into the strbuf "buf", overwriting
- * any existing contents. A pointer to buf->buf is returned as a convenience.
- *
- * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx"
- */
-extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);
-
 /*
  * Create a pack .keep file named "name" (which should generally be the output
  * of odb_pack_name). Returns a file descriptor opened for writing, or -1 on
pack.c new file +23 -0
new file mode 100644
index 000000000..0d191dfd6
@@ -0,0 +1,23 @@
+#include "cache.h"
+
+char *odb_pack_name(struct strbuf *buf,
+		    const unsigned char *sha1,
+		    const char *ext)
+{
+	strbuf_reset(buf);
+	strbuf_addf(buf, "%s/pack/pack-%s.%s", get_object_directory(),
+		    sha1_to_hex(sha1), ext);
+	return buf->buf;
+}
+
+char *sha1_pack_name(const unsigned char *sha1)
+{
+	static struct strbuf buf = STRBUF_INIT;
+	return odb_pack_name(&buf, sha1, "pack");
+}
+
+char *sha1_pack_index_name(const unsigned char *sha1)
+{
+	static struct strbuf buf = STRBUF_INIT;
+	return odb_pack_name(&buf, sha1, "idx");
+}
pack.h +23 -0
index 8294341af..63bfde00c 100644
@@ -101,4 +101,27 @@ extern int read_pack_header(int fd, struct pack_header *);
 extern struct sha1file *create_tmp_packfile(char **pack_tmp_name);
 extern void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);
 
+/*
+ * Generate the filename to be used for a pack file with checksum "sha1" and
+ * extension "ext". The result is written into the strbuf "buf", overwriting
+ * any existing contents. A pointer to buf->buf is returned as a convenience.
+ *
+ * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx"
+ */
+extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);
+
+/*
+ * Return the name of the (local) packfile with the specified sha1 in
+ * its name.  The return value is a pointer to memory that is
+ * overwritten each time this function is called.
+ */
+extern char *sha1_pack_name(const unsigned char *sha1);
+
+/*
+ * Return the name of the (local) pack index file with the specified
+ * sha1 in its name.  The return value is a pointer to memory that is
+ * overwritten each time this function is called.
+ */
+extern char *sha1_pack_index_name(const unsigned char *sha1);
+
 #endif
sha1_file.c +0 -22
index b60ae15f7..7e511ce9e 100644
@@ -278,28 +278,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
- char *odb_pack_name(struct strbuf *buf,
-		     const unsigned char *sha1,
-		     const char *ext)
-{
-	strbuf_reset(buf);
-	strbuf_addf(buf, "%s/pack/pack-%s.%s", get_object_directory(),
-		    sha1_to_hex(sha1), ext);
-	return buf->buf;
-}
-
-char *sha1_pack_name(const unsigned char *sha1)
-{
-	static struct strbuf buf = STRBUF_INIT;
-	return odb_pack_name(&buf, sha1, "pack");
-}
-
-char *sha1_pack_index_name(const unsigned char *sha1)
-{
-	static struct strbuf buf = STRBUF_INIT;
-	return odb_pack_name(&buf, sha1, "idx");
-}
-
 struct alternate_object_database *alt_odb_list;
 static struct alternate_object_database **alt_odb_tail;
 
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#2 lore raw

Re: [RFC PATCH 01/10] pack: move pack name-related functions

Quoted 7 lines · Jonathan Tan

Currently, sha1_file.c and cache.h contain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.

Create a new file, pack.c, to hold all packfile-related functions currently in sha1_file.c, and designate pack.h to hold these packfile-related functions.

There are also packed refs, so one could (like I did) think that pack.c is for generic packing of things, maybe packfile.c would be more clear?

#3 lore raw

Re: [RFC PATCH 01/10] pack: move pack name-related functions

Quoted 5 lines · On Tue, 8 Aug 2017 13:36:24 -0700 Stefan Beller <[email protected]> wrote:
Quoted 4 lines · Jonathan Tan

Currently, sha1_file.c and cache.h contain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.

Quoted 7 lines
Quoted 3 lines

Create a new file, pack.c, to hold all packfile-related functions currently in sha1_file.c, and designate pack.h to hold these packfile-related functions.

There are also packed refs, so one could (like I did) think that pack.c is for generic packing of things, maybe packfile.c would be more clear?

Good point. I'll use packfile.c and packfile.h in the next version.

#4 lore raw

Re: [RFC PATCH 01/10] pack: move pack name-related functions

Quoted 8 lines · Jonathan Tan
Quoted 4 lines · On Tue, 8 Aug 2017 13:36:24 -0700 Stefan Beller <[email protected]> wrote:

There are also packed refs, so one could (like I did) think that pack.c is for generic packing of things, maybe packfile.c would be more clear?

Good point. I'll use packfile.c and packfile.h in the next version.

It looks like you used "packfile.c" and "pack.h" in v2. Is there a reason why it's not using "packfile.h"?

#5 lore raw

Re: [RFC PATCH 01/10] pack: move pack name-related functions

Quoted 4 lines · On Wed, 9 Aug 2017 14:00:40 +0200 Christian Couder…
Quoted 3 lines · Jonathan Tan
Quoted 1 line · On Tue, 8 Aug 2017 13:36:24 -0700 Stefan Beller <[email protected]> wrote:
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

There are also packed refs, so one could (like I did) think that pack.c is for generic packing of things, maybe packfile.c would be more clear?

Good point. I'll use packfile.c and packfile.h in the next version.

It looks like you used "packfile.c" and "pack.h" in v2. Is there a reason why it's not using "packfile.h"?

Ah, I forgot to mention this in the cover letter. I thought that one header was sufficient to cover all pack-related things, so if we wanted to know which files used pack-related things, we would only need to search for one string instead of two. Also, the division between "pack.h" and the hypothetical "packfile.h" was not so clear to me.

#6 lore raw

Re: [RFC PATCH 01/10] pack: move pack name-related functions

Quoted 13 lines · Jonathan Tan
Quoted 10 lines · On Wed, 9 Aug 2017 14:00:40 +0200 Christian Couder…
Quoted 8 lines · Jonathan Tan
Quoted 4 lines · On Tue, 8 Aug 2017 13:36:24 -0700 Stefan Beller <[email protected]> wrote:

There are also packed refs, so one could (like I did) think that pack.c is for generic packing of things, maybe packfile.c would be more clear?

Good point. I'll use packfile.c and packfile.h in the next version.

Quoted 9 lines
Quoted 2 lines

It looks like you used "packfile.c" and "pack.h" in v2. Is there a reason why it's not using "packfile.h"?

Ah, I forgot to mention this in the cover letter. I thought that one header was sufficient to cover all pack-related things, so if we wanted to know which files used pack-related things, we would only need to search for one string instead of two. Also, the division between "pack.h" and the hypothetical "packfile.h" was not so clear to me.

I prefer having source and the header files that export the functions have matching names to make it easy to find them. I would prefer packfile.h vs pack.h myself.

#7 lore raw

Re: [RFC PATCH 01/10] pack: move pack name-related functions

Quoted 11 lines · Ben Peart
Quoted 5 lines · Jonathan Tan

Ah, I forgot to mention this in the cover letter. I thought that one header was sufficient to cover all pack-related things, so if we wanted to know which files used pack-related things, we would only need to search for one string instead of two. Also, the division between "pack.h" and the hypothetical "packfile.h" was not so clear to me.

I prefer having source and the header files that export the functions have matching names to make it easy to find them. I would prefer packfile.h vs pack.h myself.

Meaning "If we have packfile.c, packfile.h is preferrable over pack.h"? I tend to agree with that.

#8 lore raw

Re: [RFC PATCH 01/10] pack: move pack name-related functions

Quoted 10 lines · On Fri, 11 Aug 2017 14:34:27 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 8 lines · Ben Peart
Quoted 5 lines · Jonathan Tan

Ah, I forgot to mention this in the cover letter. I thought that one header was sufficient to cover all pack-related things, so if we wanted to know which files used pack-related things, we would only need to search for one string instead of two. Also, the division between "pack.h" and the hypothetical "packfile.h" was not so clear to me.

Quoted 6 lines
Quoted 3 lines

I prefer having source and the header files that export the functions have matching names to make it easy to find them. I would prefer packfile.h vs pack.h myself.

Meaning "If we have packfile.c, packfile.h is preferrable over pack.h"? I tend to agree with that.

Fair enough - I've changed it so that the functions now go into packfile.h. I'll send it out once I know what to base it on (at least jt/sha1-file-cleanup, and a few more branches that also modify sha1_file.c).

#9 lore raw

RFC PATCH 2/10 pack: move static state variables

In reply to Jonathan Tan

sha1_file.c declares some static variables that store packfile-related state. Move them to pack.c.

They are temporarily made global, but subsequent commits will restore their scope back to static.

---
 pack.c      | 14 ++++++++++++++
 pack.h      |  9 +++++++++
 sha1_file.c | 13 -------------
 3 files changed, 23 insertions(+), 13 deletions(-)
pack.c +14 -0
index 0d191dfd6..0f46e0617 100644
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "mru.h"
 
 char *odb_pack_name(struct strbuf *buf,
 		    const unsigned char *sha1,
@@ -21,3 +22,16 @@ char *sha1_pack_index_name(const unsigned char *sha1)
 	static struct strbuf buf = STRBUF_INIT;
 	return odb_pack_name(&buf, sha1, "idx");
 }
+
+unsigned int pack_used_ctr;
+unsigned int pack_mmap_calls;
+unsigned int peak_pack_open_windows;
+unsigned int pack_open_windows;
+unsigned int pack_open_fds;
+unsigned int pack_max_fds;
+size_t peak_pack_mapped;
+size_t pack_mapped;
+struct packed_git *packed_git;
+
+static struct mru packed_git_mru_storage;
+struct mru *packed_git_mru = &packed_git_mru_storage;
pack.h +9 -0
index 63bfde00c..7fcd45f7b 100644
@@ -124,4 +124,13 @@ extern char *sha1_pack_name(const unsigned char *sha1);
  */
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 
+extern unsigned int pack_used_ctr;
+extern unsigned int pack_mmap_calls;
+extern unsigned int peak_pack_open_windows;
+extern unsigned int pack_open_windows;
+extern unsigned int pack_open_fds;
+extern unsigned int pack_max_fds;
+extern size_t peak_pack_mapped;
+extern size_t pack_mapped;
+
 #endif
sha1_file.c +0 -13
index 7e511ce9e..4d95e21eb 100644
@@ -682,19 +682,6 @@ static int has_loose_object(const unsigned char *sha1)
 	return check_and_freshen(sha1, 0);
 }
 
-static unsigned int pack_used_ctr;
-static unsigned int pack_mmap_calls;
-static unsigned int peak_pack_open_windows;
-static unsigned int pack_open_windows;
-static unsigned int pack_open_fds;
-static unsigned int pack_max_fds;
-static size_t peak_pack_mapped;
-static size_t pack_mapped;
-struct packed_git *packed_git;
-
-static struct mru packed_git_mru_storage;
-struct mru *packed_git_mru = &packed_git_mru_storage;
-
 void pack_report(void)
 {
 	fprintf(stderr,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#10 lore raw

RFC PATCH 3/10 pack: move pack_report()

In reply to Jonathan Tan
---
 cache.h     |  2 --
 pack.c      | 24 ++++++++++++++++++++++++
 pack.h      |  2 ++
 sha1_file.c | 24 ------------------------
 4 files changed, 26 insertions(+), 26 deletions(-)
cache.h +0 -2
index 1f0f47819..c7f802e4a 100644
@@ -1624,8 +1624,6 @@ unsigned long approximate_object_count(void);
 extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
 					 struct packed_git *packs);
 
-extern void pack_report(void);
-
 /*
  * Create a temporary file rooted in the object database directory, or
  * die on failure. The filename is taken from "pattern", which should have the
pack.c +24 -0
index 0f46e0617..60d9fc3b0 100644
@@ -35,3 +35,27 @@ struct packed_git *packed_git;
 
 static struct mru packed_git_mru_storage;
 struct mru *packed_git_mru = &packed_git_mru_storage;
+
+#define SZ_FMT PRIuMAX
+static inline uintmax_t sz_fmt(size_t s) { return s; }
+
+void pack_report(void)
+{
+	fprintf(stderr,
+		"pack_report: getpagesize()            = %10" SZ_FMT "\n"
+		"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n"
+		"pack_report: core.packedGitLimit      = %10" SZ_FMT "\n",
+		sz_fmt(getpagesize()),
+		sz_fmt(packed_git_window_size),
+		sz_fmt(packed_git_limit));
+	fprintf(stderr,
+		"pack_report: pack_used_ctr            = %10u\n"
+		"pack_report: pack_mmap_calls          = %10u\n"
+		"pack_report: pack_open_windows        = %10u / %10u\n"
+		"pack_report: pack_mapped              = "
+			"%10" SZ_FMT " / %10" SZ_FMT "\n",
+		pack_used_ctr,
+		pack_mmap_calls,
+		pack_open_windows, peak_pack_open_windows,
+		sz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));
+}
pack.h +2 -0
index 7fcd45f7b..6098bfe40 100644
@@ -133,4 +133,6 @@ extern unsigned int pack_max_fds;
 extern size_t peak_pack_mapped;
 extern size_t pack_mapped;
 
+extern void pack_report(void);
+
 #endif
sha1_file.c +0 -24
index 4d95e21eb..0de39f480 100644
@@ -29,9 +29,6 @@
 #include "mergesort.h"
 #include "quote.h"
 
-#define SZ_FMT PRIuMAX
-static inline uintmax_t sz_fmt(size_t s) { return s; }
-
 const unsigned char null_sha1[20];
 const struct object_id null_oid;
 const struct object_id empty_tree_oid = {
@@ -682,27 +679,6 @@ static int has_loose_object(const unsigned char *sha1)
 	return check_and_freshen(sha1, 0);
 }
 
-void pack_report(void)
-{
-	fprintf(stderr,
-		"pack_report: getpagesize()            = %10" SZ_FMT "\n"
-		"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n"
-		"pack_report: core.packedGitLimit      = %10" SZ_FMT "\n",
-		sz_fmt(getpagesize()),
-		sz_fmt(packed_git_window_size),
-		sz_fmt(packed_git_limit));
-	fprintf(stderr,
-		"pack_report: pack_used_ctr            = %10u\n"
-		"pack_report: pack_mmap_calls          = %10u\n"
-		"pack_report: pack_open_windows        = %10u / %10u\n"
-		"pack_report: pack_mapped              = "
-			"%10" SZ_FMT " / %10" SZ_FMT "\n",
-		pack_used_ctr,
-		pack_mmap_calls,
-		pack_open_windows, peak_pack_open_windows,
-		sz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));
-}
-
 /*
  * Open and mmap the index file at path, perform a couple of
  * consistency checks, then record its information to p.  Return 0 on
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#11 lore raw

RFC PATCH 4/10 pack: move open_pack_index(), parse_pack_index()

In reply to Jonathan Tan
---
 builtin/count-objects.c |   1 +
 cache.h                 |   8 ---
 pack.c                  | 149 ++++++++++++++++++++++++++++++++++++++++++++++++
 pack.h                  |   8 +++
 sha1_file.c             | 140 ---------------------------------------------
 sha1_name.c             |   1 +
 6 files changed, 159 insertions(+), 148 deletions(-)
builtin/count-objects.c +1 -0
index 1d82e61f2..185d3190a 100644
@@ -10,6 +10,7 @@
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
+#include "pack.h"
 
 static unsigned long garbage;
 static off_t size_garbage;
cache.h +0 -8
index c7f802e4a..5d6839525 100644
@@ -1603,8 +1603,6 @@ struct pack_entry {
 	struct packed_git *p;
 };
 
-extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
-
 /* A hook to report invalid files in pack directory */
 #define PACKDIR_FILE_PACK 1
 #define PACKDIR_FILE_IDX 2
@@ -1639,12 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * mmap the index file for the specified packfile (if it is not
- * already mmapped).  Return 0 on success.
- */
-extern int open_pack_index(struct packed_git *);
-
 /*
  * munmap the index file for the specified packfile (if it is
  * currently mmapped).
pack.c +149 -0

157 lines not on this page · raw message

pack.h +8 -0
index 6098bfe40..5be0ed42a 100644
@@ -135,4 +135,12 @@ extern size_t pack_mapped;
 
 extern void pack_report(void);
 
+/*
+ * mmap the index file for the specified packfile (if it is not
+ * already mmapped).  Return 0 on success.
+ */
+extern int open_pack_index(struct packed_git *);
+
+extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
+
 #endif
sha1_file.c +0 -140

152 lines not on this page · raw message

sha1_name.c +1 -0
index 74fcb6d78..28b7c9fd8 100644
@@ -9,6 +9,7 @@
 #include "remote.h"
 #include "dir.h"
 #include "sha1-array.h"
+#include "pack.h"
 
 static int get_sha1_oneline(const char *, unsigned char *, struct commit_list *);
 
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#12 lore raw

Re: [RFC PATCH 04/10] pack: move open_pack_index(), parse_pack_index()

Quoted 9 lines · Jonathan Tan
---
 builtin/count-objects.c |   1 +
 cache.h                 |   8 ---
 pack.c                  | 149 ++++++++++++++++++++++++++++++++++++++++++++++++
 pack.h                  |   8 +++
 sha1_file.c             | 140 ---------------------------------------------
 sha1_name.c             |   1 +
 6 files changed, 159 insertions(+), 148 deletions(-)

This patch is a bit strange...

Quoted 18 lines
pack.c +13 -0
index 60d9fc3b0..6edc43228 100644
...
+static struct packed_git *alloc_packed_git(int extra)
+{
+	struct packed_git *p = xmalloc(st_add(sizeof(*p), extra));
+	memset(p, 0, sizeof(*p));
+	p->pack_fd = -1;
+	return p;
+}
+
+struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)
+{
+	const char *path = sha1_pack_name(sha1);
+	size_t alloc = st_add(strlen(path), 1);
+	struct packed_git *p = alloc_packed_git(alloc);

pack.c void pack_report(void) · line 210

Quoted 10 lines
pack.c +10 -0
+
+	memcpy(p->pack_name, path, alloc); /* includes NUL */
+	hashcpy(p->sha1, sha1);
+	if (check_packed_git_idx(idx_path, p)) {
+		free(p);
+		return NULL;
+	}
+
+	return p;
+}

We see these two functions appear in pack.c

Quoted 15 lines
sha1_file.c +0 -6
index 0de39f480..2e414f5f5 100644
...
@@ -1300,22 +1176,6 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
 	return p;
 }
 
-struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)
-{
-	const char *path = sha1_pack_name(sha1);
-	size_t alloc = st_add(strlen(path), 1);
-	struct packed_git *p = alloc_packed_git(alloc);
-

sha1_file.c struct packed_git *add_packed_git(const char *path, size_t path_len, int local) · line 1318

Quoted 10 lines
sha1_file.c +0 -10
-	memcpy(p->pack_name, path, alloc); /* includes NUL */
-	hashcpy(p->sha1, sha1);
-	if (check_packed_git_idx(idx_path, p)) {
-		free(p);
-		return NULL;
-	}
-
-	return p;
-}
-

And we see parse_pack_index() came from sha1_file.c

But where did alloc_packed_git() come from? Was the patch split incorrectly or something?

When I applied the whole series and did

git blame -s -w -M -C -C master.. pack.c

expecting that pretty much everything has come from sha1_file.c but noticed that some lines were actually blamed to a version of pack.c and these functions were among them.

#13 lore raw

Re: [RFC PATCH 04/10] pack: move open_pack_index(), parse_pack_index()

Quoted 76 lines · On Tue, 08 Aug 2017 13:19:23 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 9 lines · Jonathan Tan
---
 builtin/count-objects.c |   1 +
 cache.h                 |   8 ---
 pack.c                  | 149 ++++++++++++++++++++++++++++++++++++++++++++++++
 pack.h                  |   8 +++
 sha1_file.c             | 140 ---------------------------------------------
 sha1_name.c             |   1 +
 6 files changed, 159 insertions(+), 148 deletions(-)

This patch is a bit strange...

Quoted 18 lines
pack.c +13 -0
index 60d9fc3b0..6edc43228 100644
...
+static struct packed_git *alloc_packed_git(int extra)
+{
+	struct packed_git *p = xmalloc(st_add(sizeof(*p), extra));
+	memset(p, 0, sizeof(*p));
+	p->pack_fd = -1;
+	return p;
+}
+
+struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)
+{
+	const char *path = sha1_pack_name(sha1);
+	size_t alloc = st_add(strlen(path), 1);
+	struct packed_git *p = alloc_packed_git(alloc);

pack.c void pack_report(void) · line 210

Quoted 10 lines
pack.c +10 -0
+
+	memcpy(p->pack_name, path, alloc); /* includes NUL */
+	hashcpy(p->sha1, sha1);
+	if (check_packed_git_idx(idx_path, p)) {
+		free(p);
+		return NULL;
+	}
+
+	return p;
+}

We see these two functions appear in pack.c

Quoted 15 lines
sha1_file.c +0 -6
index 0de39f480..2e414f5f5 100644
...
@@ -1300,22 +1176,6 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
 	return p;
 }
 
-struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path)
-{
-	const char *path = sha1_pack_name(sha1);
-	size_t alloc = st_add(strlen(path), 1);
-	struct packed_git *p = alloc_packed_git(alloc);
-

sha1_file.c struct packed_git *add_packed_git(const char *path, size_t path_len, int local) · line 1318

Quoted 10 lines
sha1_file.c +0 -10
-	memcpy(p->pack_name, path, alloc); /* includes NUL */
-	hashcpy(p->sha1, sha1);
-	if (check_packed_git_idx(idx_path, p)) {
-		free(p);
-		return NULL;
-	}
-
-	return p;
-}
-

And we see parse_pack_index() came from sha1_file.c

But where did alloc_packed_git() come from? Was the patch split incorrectly or something?

Quoted 7 lines

When I applied the whole series and did

git blame -s -w -M -C -C master.. pack.c

expecting that pretty much everything has come from sha1_file.c but noticed that some lines were actually blamed to a version of pack.c and these functions were among them.

alloc_packed_git() in pack.c is a duplicate of the function of the same name in sha1_file.c in this patch, because at this patch, there are still functions in both files using this function. A subsequent patch in this patch set will remove it from pack.c.

I'll add a note explaining this to this patch in the next version.

#14 lore raw

RFC PATCH 5/10 pack: move release_pack_memory()

In reply to Jonathan Tan

The function unuse_one_window() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.

---
 git-compat-util.h |  2 --
 pack.c            | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 pack.h            |  4 ++++
 sha1_file.c       | 49 -------------------------------------------------
 4 files changed, 53 insertions(+), 51 deletions(-)
git-compat-util.h +0 -2
index db9c22de7..201056e2d 100644
@@ -749,8 +749,6 @@ const char *inet_ntop(int af, const void *src, char *dst, size_t size);
 extern int git_atexit(void (*handler)(void));
 #endif
 
-extern void release_pack_memory(size_t);
-
 typedef void (*try_to_free_t)(size_t);
 extern try_to_free_t set_try_to_free_routine(try_to_free_t);
 
pack.c +49 -0

52 lines not on this page · raw message

pack.h +4 -0
index 5be0ed42a..c16220586 100644
@@ -143,4 +143,8 @@ extern int open_pack_index(struct packed_git *);
 
 extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
 
+extern int unuse_one_window(struct packed_git *current);
+
+extern void release_pack_memory(size_t);
+
 #endif
sha1_file.c +0 -49

55 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#15 lore raw

RFC PATCH 6/10 pack: move pack-closing functions

In reply to Jonathan Tan

The function close_pack_fd() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.

---
In doing this, I discovered that some builtins close the packs even though they, in theory, should not know anything about how objects are stored. Can we remove those calls? (The tests pass with those calls removed.)

---
 builtin/am.c    |  1 +
 builtin/clone.c |  1 +
 builtin/fetch.c |  1 +
 builtin/merge.c |  1 +
 cache.h         |  8 --------
 pack.c          | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 pack.h          |  9 +++++++++
 sha1_file.c     | 55 -------------------------------------------------------
 8 files changed, 67 insertions(+), 63 deletions(-)
builtin/am.c +1 -0
index c973bd96d..c38dd10a3 100644
@@ -31,6 +31,7 @@
 #include "mailinfo.h"
 #include "apply.h"
 #include "string-list.h"
+#include "pack.h"
 
 /**
  * Returns 1 if the file is empty or does not exist, 0 otherwise.
builtin/clone.c +1 -0
index 08b5cc433..53410a45d 100644
@@ -25,6 +25,7 @@
 #include "remote.h"
 #include "run-command.h"
 #include "connected.h"
+#include "pack.h"
 
 /*
  * Overall FIXMEs:
builtin/fetch.c +1 -0
index c87e59f3b..196a3bfc4 100644
@@ -17,6 +17,7 @@
 #include "connected.h"
 #include "argv-array.h"
 #include "utf8.h"
+#include "pack.h"
 
 static const char * const builtin_fetch_usage[] = {
 	N_("git fetch [<options>] [<repository> [<refspec>...]]"),
builtin/merge.c +1 -0
index 900bafdb4..9cff4b276 100644
@@ -32,6 +32,7 @@
 #include "gpg-interface.h"
 #include "sequencer.h"
 #include "string-list.h"
+#include "pack.h"
 
 #define DEFAULT_TWOHEAD (1<<0)
 #define DEFAULT_OCTOPUS (1<<1)
cache.h +0 -8
index 5d6839525..25a21a61f 100644
@@ -1637,15 +1637,7 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * munmap the index file for the specified packfile (if it is
- * currently mmapped).
- */
-extern void close_pack_index(struct packed_git *);
-
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
-extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
pack.c +54 -0

57 lines not on this page · raw message

pack.h +9 -0
index c16220586..fd4668528 100644
@@ -147,4 +147,13 @@ extern int unuse_one_window(struct packed_git *current);
 
 extern void release_pack_memory(size_t);
 
+extern void close_pack_windows(struct packed_git *);
+extern int close_pack_fd(struct packed_git *);
+/*
+ * munmap the index file for the specified packfile (if it is
+ * currently mmapped).
+ */
+extern void close_pack_index(struct packed_git *);
+extern void close_all_packs(void);
+
 #endif
sha1_file.c +0 -55

67 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#16 lore raw

RFC PATCH 7/10 pack: move use_pack()

In reply to Jonathan Tan

open_packed_git is made global.

---
Unlike the other commits where variables and functions are made global and then remade static, open_packed_git is not remade static (because a function in sha1_file.c still uses it).

---
 cache.h     |   1 -
 pack.c      | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 pack.h      |  14 +--
 sha1_file.c | 285 --------------------------------------------------------
 streaming.c |   1 +
 5 files changed, 299 insertions(+), 305 deletions(-)
cache.h +0 -1
index 25a21a61f..dd9f9a9ae 100644
@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
pack.c +294 -9

325 lines not on this page · raw message

pack.h +4 -10
index fd4668528..bf2b99bf9 100644
@@ -124,14 +124,7 @@ extern char *sha1_pack_name(const unsigned char *sha1);
  */
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 
-extern unsigned int pack_used_ctr;
-extern unsigned int pack_mmap_calls;
-extern unsigned int peak_pack_open_windows;
-extern unsigned int pack_open_windows;
 extern unsigned int pack_open_fds;
-extern unsigned int pack_max_fds;
-extern size_t peak_pack_mapped;
-extern size_t pack_mapped;
 
 extern void pack_report(void);
 
@@ -143,12 +136,9 @@ extern int open_pack_index(struct packed_git *);
 
 extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
 
-extern int unuse_one_window(struct packed_git *current);
-
 extern void release_pack_memory(size_t);
 
 extern void close_pack_windows(struct packed_git *);
-extern int close_pack_fd(struct packed_git *);
 /*
  * munmap the index file for the specified packfile (if it is
  * currently mmapped).
@@ -156,4 +146,8 @@ extern int close_pack_fd(struct packed_git *);
 extern void close_pack_index(struct packed_git *);
 extern void close_all_packs(void);
 
+extern int open_packed_git(struct packed_git *p);
+
+extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
+
 #endif
sha1_file.c +0 -285

297 lines not on this page · raw message

streaming.c +1 -0
index 9afa66b8b..f657018cf 100644
@@ -3,6 +3,7 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "pack.h"
 
 enum input_source {
 	stream_error = -1,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#17 lore raw

RFC PATCH 8/10 pack: move unuse_pack()

In reply to Jonathan Tan
---
 cache.h     | 1 -
 pack.c      | 9 +++++++++
 pack.h      | 1 +
 sha1_file.c | 9 ---------
 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -1
index dd9f9a9ae..4812f3a63 100644
@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
pack.c +9 -0
index 85cb65558..93526ea7b 100644
@@ -596,3 +596,12 @@ unsigned char *use_pack(struct packed_git *p,
 		*left = win->len - xsize_t(offset);
 	return win->base + offset;
 }
+
+void unuse_pack(struct pack_window **w_cursor)
+{
+	struct pack_window *w = *w_cursor;
+	if (w) {
+		w->inuse_cnt--;
+		*w_cursor = NULL;
+	}
+}
pack.h +1 -0
index bf2b99bf9..3876e9ae6 100644
@@ -149,5 +149,6 @@ extern void close_all_packs(void);
 extern int open_packed_git(struct packed_git *p);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
+extern void unuse_pack(struct pack_window **);
 
 #endif
sha1_file.c +0 -9
index 8f17a07e9..12501ef06 100644
@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
-void unuse_pack(struct pack_window **w_cursor)
-{
-	struct pack_window *w = *w_cursor;
-	if (w) {
-		w->inuse_cnt--;
-		*w_cursor = NULL;
-	}
-}
-
 static struct packed_git *alloc_packed_git(int extra)
 {
 	struct packed_git *p = xmalloc(st_add(sizeof(*p), extra));
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#18 lore raw

RFC PATCH 9/10 pack: move add_packed_git()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 connected.c |  1 +
 pack.c      | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 pack.h      |  1 +
 sha1_file.c | 61 -------------------------------------------------------------
 5 files changed, 55 insertions(+), 62 deletions(-)
cache.h +0 -1
index 4812f3a63..bf93477e8 100644
@@ -1638,7 +1638,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
 extern int odb_pack_keep(const char *name);
 
 extern void clear_delta_base_cache(void);
-extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
 /*
  * Make sure that a pointer access into an mmap'd index file is within bounds,
connected.c +1 -0
index 136c2ac16..3e3f0148c 100644
@@ -3,6 +3,7 @@
 #include "sigchain.h"
 #include "connected.h"
 #include "transport.h"
+#include "pack.h"
 
 /*
  * If we feed all the commits we want to verify to this command
pack.c +53 -0

56 lines not on this page · raw message

pack.h +1 -0
index 3876e9ae6..c1f3ff32d 100644
@@ -150,5 +150,6 @@ extern int open_packed_git(struct packed_git *p);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void unuse_pack(struct pack_window **);
+extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
 #endif
sha1_file.c +0 -61

67 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#19 lore raw

RFC PATCH 10/10 pack: move install_packed_git()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 pack.c      | 11 ++++++++++-
 pack.h      |  4 ++--
 sha1_file.c |  9 ---------
 4 files changed, 12 insertions(+), 13 deletions(-)
cache.h +0 -1
index bf93477e8..41562dc0b 100644
@@ -1611,7 +1611,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
 extern void prepare_packed_git(void);
 extern void reprepare_packed_git(void);
-extern void install_packed_git(struct packed_git *pack);
 
 /*
  * Give a rough count of objects in the repository. This sacrifices accuracy
pack.c +10 -1
index efe0ed3e8..4eb65e460 100644
@@ -28,7 +28,7 @@ static unsigned int pack_used_ctr;
 static unsigned int pack_mmap_calls;
 static unsigned int peak_pack_open_windows;
 static unsigned int pack_open_windows;
-unsigned int pack_open_fds;
+static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
@@ -658,3 +658,12 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
 		hashclr(p->sha1);
 	return p;
 }
+
+void install_packed_git(struct packed_git *pack)
+{
+	if (pack->pack_fd != -1)
+		pack_open_fds++;
+
+	pack->next = packed_git;
+	packed_git = pack;
+}
pack.h +2 -2
index c1f3ff32d..576c4fc7c 100644
@@ -124,8 +124,6 @@ extern char *sha1_pack_name(const unsigned char *sha1);
  */
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 
-extern unsigned int pack_open_fds;
-
 extern void pack_report(void);
 
 /*
@@ -152,4 +150,6 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t
 extern void unuse_pack(struct pack_window **);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
+extern void install_packed_git(struct packed_git *pack);
+
 #endif
sha1_file.c +0 -9
index 7f12b1ee0..b956ca0c9 100644
@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
-void install_packed_git(struct packed_git *pack)
-{
-	if (pack->pack_fd != -1)
-		pack_open_fds++;
-
-	pack->next = packed_git;
-	packed_git = pack;
-}
-
 void (*report_garbage)(unsigned seen_bits, const char *path);
 
 static void report_helper(const struct string_list *list,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#20 lore raw
In reply to Jonathan Tan
Quoted 8 lines · Jonathan Tan

While investigating annotating packfiles and loose objects to support connectivity checks in partial clones [1], I decided to make the effort to separate packfile-related code from sha1_file.c into its own file, to make it easier to both code such changes and review them. Here is the beginning of those efforts.

Is this something worth doing, and if yes, is this in the right direction?

Overall I think it is a good idea to slim down sha1_file.c *if* we can keep the exposed surface area small enough.

I wonder if the names "pack.[ch]" communicate well that these are "object access layer that is about reading from packfiles". The writer side is called "pack-objects.[ch]".

This may have to make some symbols that used to be private to the "object access" layer, which was what sha1_file.c was about, global symbols. After moving things around, do we end up exposing too many implementation details to the world outside the "object access" layer? I'd assume they are limited to the resulting pack.h and it would be OK as long as nobody other than sha1_file.c and pack.c would inculde it, though.

Thanks.

Quoted 32 lines

[1] https://public-inbox.org/git/[email protected]/

Jonathan Tan (10):
pack: move pack name-related functions
pack: move static state variables
pack: move pack_report()
pack: move open_pack_index(), parse_pack_index()
pack: move release_pack_memory()
pack: move pack-closing functions
pack: move use_pack()
pack: move unuse_pack()
pack: move add_packed_git()
pack: move install_packed_git()

 Makefile                 |   1 +
 builtin/am.c             |   1 +
 builtin/clone.c          |   1 +
 builtin/count-objects.c  |   1 +
 builtin/fetch.c          |   1 +
 builtin/merge.c          |   1 +
 builtin/pack-redundant.c |   1 +
 cache.h                  |  45 ----
 connected.c              |   1 +
 git-compat-util.h        |   2 -
 pack.c                   | 669 +++++++++++++++++++++++++++++++++++++++++++++++
 pack.h                   |  51 ++++
 sha1_file.c              | 667 ----------------------------------------------
 sha1_name.c              |   1 +
 streaming.c              |   1 +
 15 files changed, 730 insertions(+), 714 deletions(-)
 create mode 100644 pack.c
#21 lore raw
Quoted 8 lines · On Tue, 08 Aug 2017 13:05:05 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 6 lines · Jonathan Tan

While investigating annotating packfiles and loose objects to support connectivity checks in partial clones [1], I decided to make the effort to separate packfile-related code from sha1_file.c into its own file, to make it easier to both code such changes and review them. Here is the beginning of those efforts.

Quoted 5 lines
Quoted 2 lines

Is this something worth doing, and if yes, is this in the right direction?

Overall I think it is a good idea to slim down sha1_file.c *if* we can keep the exposed surface area small enough.

What do you mean by "keep the exposed surface area small enough"? If you mean the total number of exposed functions in sha1_file and pack (once everything is done), I think it will be almost the same as that currently in sha1_file.

find_pack_entry() and has_packed_and_bad() (not yet in this patch set) might need to be changed from static to global, but those are the only 2 I can think of. Anyway, I'll report the functions that need to be changed from static to global at the end.

During this patch set, there might be some functions that need to be temporarily made global, but those are reverted to static in the end.

Quoted 3 lines

I wonder if the names "pack.[ch]" communicate well that these are "object access layer that is about reading from packfiles". The writer side is called "pack-objects.[ch]".

This file will end up being slightly broader than reading from packfiles
- in particular, things like pack_report() (reporting some statistics not only on the in-repo packfiles themselves) and parse_pack_index() (which parses an idx file obtained through http) are there too. Hence the generic name, but I agree that there might be a better name (or better set of names).

Quoted 7 lines

This may have to make some symbols that used to be private to the "object access" layer, which was what sha1_file.c was about, global symbols. After moving things around, do we end up exposing too many implementation details to the world outside the "object access" layer? I'd assume they are limited to the resulting pack.h and it would be OK as long as nobody other than sha1_file.c and pack.c would inculde it, though.

As stated above, I don't think so, but I'll make a list of the functions needing to be made global.

#22 lore raw
Quoted 7 lines · Jonathan Tan

What do you mean by "keep the exposed surface area small enough"? If you mean the total number of exposed functions in sha1_file and pack (once everything is done), I think it will be almost the same as that currently in sha1_file.
...
During this patch set, there might be some functions that need to be temporarily made global, but those are reverted to static in the end.

That is exactly what I meant.

Quoted 2 lines

As stated above, I don't think so, but I'll make a list of the functions needing to be made global.

Good.

Thanks.

#23 lore raw

v2 cover Move exported packfile funcs to its own file

In reply to Jonathan Tan

Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like freshen_packed_object() that are used only by non-pack-specific functions are not moved.

In the end, 3 functions needed to be made global. They are find_pack_entry(), mark_bad_packed_object(), and has_packed_and_bad().

Of the 3, find_pack_entry() is probably legitimately promoted. But I think that the latter two functions needing to be accessed from sha1_file.c points to a design that could be improved - they are only used when packed_object_info() detects corruption, and used for marking as bad and printing messages to the user respectively, which packed_object_info() should probably do itself. But I have not made this change in this patch set.

(Other than the 3 functions above, there are some variables and functions that are temporarily made global, but reduced back to static when the wide scope is no longer needed.)

Jonathan Tan (25):
pack: move pack name-related functions
pack: move static state variables
pack: move pack_report()
pack: move open_pack_index(), parse_pack_index()
pack: move release_pack_memory()
pack: move pack-closing functions
pack: move use_pack()
pack: move unuse_pack()
pack: move add_packed_git()
pack: move install_packed_git()
pack: move {,re}prepare_packed_git and approximate_object_count
pack: move unpack_object_header()
pack: move get_size_from_delta()
pack: move unpack_object_header()
sha1_file: set whence in storage-specific info fn
sha1_file: remove read_packed_sha1()
pack: move packed_object_info(), unpack_entry()
pack: move nth_packed_object_{sha1,oid}
pack: move check_pack_index_ptr(), nth_packed_object_offset()
pack: move find_pack_entry_one(), is_pack_valid()
pack: move find_sha1_pack()
pack: move find_pack_entry() and make it global
pack: move has_sha1_pack()
pack: move has_pack_index()
pack: move for_each_packed_object()

 Makefile                 |    1 +
 builtin/am.c             |    1 +
 builtin/cat-file.c       |    1 +
 builtin/clone.c          |    1 +
 builtin/count-objects.c  |    1 +
 builtin/fetch.c          |    1 +
 builtin/gc.c             |    1 +
 builtin/merge.c          |    1 +
 builtin/pack-redundant.c |    1 +
 builtin/prune-packed.c   |    1 +
 cache.h                  |  122 +--
 connected.c              |    1 +
 diff.c                   |    1 +
 git-compat-util.h        |    2 -
 http-backend.c           |    1 +
 http-push.c              |    1 +
 http-walker.c            |    1 +
 pack.h                   |  137 +++
 packfile.c               | 1905 +++++++++++++++++++++++++++++++++++
 path.c                   |    1 +
 reachable.c              |    1 +
 revision.c               |    1 +
 server-info.c            |    1 +
 sha1_file.c              | 2484 ++++++----------------------------------------
 sha1_name.c              |    1 +
 streaming.c              |    1 +
 26 files changed, 2350 insertions(+), 2321 deletions(-)
 create mode 100644 packfile.c
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#24 lore raw

PATCH v2 1/25 pack: move pack name-related functions

Currently, sha1_file.c and cache.h contain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.

Create a new file, packfile.c, to hold all packfile-related functions currently in sha1_file.c, and designate pack.h to hold these packfile-related functions.

In this commit, the pack name-related functions are moved. Subsequent commits will move the other functions.

---
 Makefile                 |  1 +
 builtin/pack-redundant.c |  1 +
 cache.h                  | 23 -----------------------
 pack.h                   | 23 +++++++++++++++++++++++
 packfile.c               | 23 +++++++++++++++++++++++
 sha1_file.c              | 22 ----------------------
 6 files changed, 48 insertions(+), 45 deletions(-)
 create mode 100644 packfile.c
Makefile +1 -0
index 461c845d3..5cdecaa17 100644
@@ -816,6 +816,7 @@ LIB_OBJS += notes-merge.o
 LIB_OBJS += notes-utils.o
 LIB_OBJS += object.o
 LIB_OBJS += oidset.o
+LIB_OBJS += packfile.o
 LIB_OBJS += pack-bitmap.o
 LIB_OBJS += pack-bitmap-write.o
 LIB_OBJS += pack-check.o
builtin/pack-redundant.c +1 -0
index cb1df1c76..df36d10e7 100644
@@ -7,6 +7,7 @@
 */
 
 #include "builtin.h"
+#include "pack.h"
 
 #define BLKSIZE 512
 
cache.h +0 -23
index 71fe09264..1f0f47819 100644
@@ -902,20 +902,6 @@ extern void check_repository_format(void);
  */
 extern const char *sha1_file_name(const unsigned char *sha1);
 
-/*
- * Return the name of the (local) packfile with the specified sha1 in
- * its name.  The return value is a pointer to memory that is
- * overwritten each time this function is called.
- */
-extern char *sha1_pack_name(const unsigned char *sha1);
-
-/*
- * Return the name of the (local) pack index file with the specified
- * sha1 in its name.  The return value is a pointer to memory that is
- * overwritten each time this function is called.
- */
-extern char *sha1_pack_index_name(const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
@@ -1648,15 +1634,6 @@ extern void pack_report(void);
  */
 extern int odb_mkstemp(struct strbuf *template, const char *pattern);
 
-/*
- * Generate the filename to be used for a pack file with checksum "sha1" and
- * extension "ext". The result is written into the strbuf "buf", overwriting
- * any existing contents. A pointer to buf->buf is returned as a convenience.
- *
- * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx"
- */
-extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);
-
 /*
  * Create a pack .keep file named "name" (which should generally be the output
  * of odb_pack_name). Returns a file descriptor opened for writing, or -1 on
pack.h +23 -0
index 8294341af..63bfde00c 100644
@@ -101,4 +101,27 @@ extern int read_pack_header(int fd, struct pack_header *);
 extern struct sha1file *create_tmp_packfile(char **pack_tmp_name);
 extern void finish_tmp_packfile(struct strbuf *name_buffer, const char *pack_tmp_name, struct pack_idx_entry **written_list, uint32_t nr_written, struct pack_idx_option *pack_idx_opts, unsigned char sha1[]);
 
+/*
+ * Generate the filename to be used for a pack file with checksum "sha1" and
+ * extension "ext". The result is written into the strbuf "buf", overwriting
+ * any existing contents. A pointer to buf->buf is returned as a convenience.
+ *
+ * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx"
+ */
+extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);
+
+/*
+ * Return the name of the (local) packfile with the specified sha1 in
+ * its name.  The return value is a pointer to memory that is
+ * overwritten each time this function is called.
+ */
+extern char *sha1_pack_name(const unsigned char *sha1);
+
+/*
+ * Return the name of the (local) pack index file with the specified
+ * sha1 in its name.  The return value is a pointer to memory that is
+ * overwritten each time this function is called.
+ */
+extern char *sha1_pack_index_name(const unsigned char *sha1);
+
 #endif
packfile.c new file +23 -0
new file mode 100644
index 000000000..0d191dfd6
@@ -0,0 +1,23 @@
+#include "cache.h"
+
+char *odb_pack_name(struct strbuf *buf,
+		    const unsigned char *sha1,
+		    const char *ext)
+{
+	strbuf_reset(buf);
+	strbuf_addf(buf, "%s/pack/pack-%s.%s", get_object_directory(),
+		    sha1_to_hex(sha1), ext);
+	return buf->buf;
+}
+
+char *sha1_pack_name(const unsigned char *sha1)
+{
+	static struct strbuf buf = STRBUF_INIT;
+	return odb_pack_name(&buf, sha1, "pack");
+}
+
+char *sha1_pack_index_name(const unsigned char *sha1)
+{
+	static struct strbuf buf = STRBUF_INIT;
+	return odb_pack_name(&buf, sha1, "idx");
+}
sha1_file.c +0 -22
index b60ae15f7..7e511ce9e 100644
@@ -278,28 +278,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
- char *odb_pack_name(struct strbuf *buf,
-		     const unsigned char *sha1,
-		     const char *ext)
-{
-	strbuf_reset(buf);
-	strbuf_addf(buf, "%s/pack/pack-%s.%s", get_object_directory(),
-		    sha1_to_hex(sha1), ext);
-	return buf->buf;
-}
-
-char *sha1_pack_name(const unsigned char *sha1)
-{
-	static struct strbuf buf = STRBUF_INIT;
-	return odb_pack_name(&buf, sha1, "pack");
-}
-
-char *sha1_pack_index_name(const unsigned char *sha1)
-{
-	static struct strbuf buf = STRBUF_INIT;
-	return odb_pack_name(&buf, sha1, "idx");
-}
-
 struct alternate_object_database *alt_odb_list;
 static struct alternate_object_database **alt_odb_tail;
 
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#25 lore raw

PATCH v2 2/25 pack: move static state variables

In reply to Jonathan Tan

sha1_file.c declares some static variables that store packfile-related state. Move them to packfile.c.

They are temporarily made global, but subsequent commits will restore their scope back to static.

---
 pack.h      |  9 +++++++++
 packfile.c  | 14 ++++++++++++++
 sha1_file.c | 13 -------------
 3 files changed, 23 insertions(+), 13 deletions(-)
pack.h +9 -0
index 63bfde00c..7fcd45f7b 100644
@@ -124,4 +124,13 @@ extern char *sha1_pack_name(const unsigned char *sha1);
  */
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 
+extern unsigned int pack_used_ctr;
+extern unsigned int pack_mmap_calls;
+extern unsigned int peak_pack_open_windows;
+extern unsigned int pack_open_windows;
+extern unsigned int pack_open_fds;
+extern unsigned int pack_max_fds;
+extern size_t peak_pack_mapped;
+extern size_t pack_mapped;
+
 #endif
packfile.c +14 -0
index 0d191dfd6..0f46e0617 100644
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "mru.h"
 
 char *odb_pack_name(struct strbuf *buf,
 		    const unsigned char *sha1,
@@ -21,3 +22,16 @@ char *sha1_pack_index_name(const unsigned char *sha1)
 	static struct strbuf buf = STRBUF_INIT;
 	return odb_pack_name(&buf, sha1, "idx");
 }
+
+unsigned int pack_used_ctr;
+unsigned int pack_mmap_calls;
+unsigned int peak_pack_open_windows;
+unsigned int pack_open_windows;
+unsigned int pack_open_fds;
+unsigned int pack_max_fds;
+size_t peak_pack_mapped;
+size_t pack_mapped;
+struct packed_git *packed_git;
+
+static struct mru packed_git_mru_storage;
+struct mru *packed_git_mru = &packed_git_mru_storage;
sha1_file.c +0 -13
index 7e511ce9e..4d95e21eb 100644
@@ -682,19 +682,6 @@ static int has_loose_object(const unsigned char *sha1)
 	return check_and_freshen(sha1, 0);
 }
 
-static unsigned int pack_used_ctr;
-static unsigned int pack_mmap_calls;
-static unsigned int peak_pack_open_windows;
-static unsigned int pack_open_windows;
-static unsigned int pack_open_fds;
-static unsigned int pack_max_fds;
-static size_t peak_pack_mapped;
-static size_t pack_mapped;
-struct packed_git *packed_git;
-
-static struct mru packed_git_mru_storage;
-struct mru *packed_git_mru = &packed_git_mru_storage;
-
 void pack_report(void)
 {
 	fprintf(stderr,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#26 lore raw

PATCH v2 3/25 pack: move pack_report()

In reply to Jonathan Tan
---
 cache.h     |  2 --
 pack.h      |  2 ++
 packfile.c  | 24 ++++++++++++++++++++++++
 sha1_file.c | 24 ------------------------
 4 files changed, 26 insertions(+), 26 deletions(-)
cache.h +0 -2
index 1f0f47819..c7f802e4a 100644
@@ -1624,8 +1624,6 @@ unsigned long approximate_object_count(void);
 extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
 					 struct packed_git *packs);
 
-extern void pack_report(void);
-
 /*
  * Create a temporary file rooted in the object database directory, or
  * die on failure. The filename is taken from "pattern", which should have the
pack.h +2 -0
index 7fcd45f7b..6098bfe40 100644
@@ -133,4 +133,6 @@ extern unsigned int pack_max_fds;
 extern size_t peak_pack_mapped;
 extern size_t pack_mapped;
 
+extern void pack_report(void);
+
 #endif
packfile.c +24 -0
index 0f46e0617..60d9fc3b0 100644
@@ -35,3 +35,27 @@ struct packed_git *packed_git;
 
 static struct mru packed_git_mru_storage;
 struct mru *packed_git_mru = &packed_git_mru_storage;
+
+#define SZ_FMT PRIuMAX
+static inline uintmax_t sz_fmt(size_t s) { return s; }
+
+void pack_report(void)
+{
+	fprintf(stderr,
+		"pack_report: getpagesize()            = %10" SZ_FMT "\n"
+		"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n"
+		"pack_report: core.packedGitLimit      = %10" SZ_FMT "\n",
+		sz_fmt(getpagesize()),
+		sz_fmt(packed_git_window_size),
+		sz_fmt(packed_git_limit));
+	fprintf(stderr,
+		"pack_report: pack_used_ctr            = %10u\n"
+		"pack_report: pack_mmap_calls          = %10u\n"
+		"pack_report: pack_open_windows        = %10u / %10u\n"
+		"pack_report: pack_mapped              = "
+			"%10" SZ_FMT " / %10" SZ_FMT "\n",
+		pack_used_ctr,
+		pack_mmap_calls,
+		pack_open_windows, peak_pack_open_windows,
+		sz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));
+}
sha1_file.c +0 -24
index 4d95e21eb..0de39f480 100644
@@ -29,9 +29,6 @@
 #include "mergesort.h"
 #include "quote.h"
 
-#define SZ_FMT PRIuMAX
-static inline uintmax_t sz_fmt(size_t s) { return s; }
-
 const unsigned char null_sha1[20];
 const struct object_id null_oid;
 const struct object_id empty_tree_oid = {
@@ -682,27 +679,6 @@ static int has_loose_object(const unsigned char *sha1)
 	return check_and_freshen(sha1, 0);
 }
 
-void pack_report(void)
-{
-	fprintf(stderr,
-		"pack_report: getpagesize()            = %10" SZ_FMT "\n"
-		"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n"
-		"pack_report: core.packedGitLimit      = %10" SZ_FMT "\n",
-		sz_fmt(getpagesize()),
-		sz_fmt(packed_git_window_size),
-		sz_fmt(packed_git_limit));
-	fprintf(stderr,
-		"pack_report: pack_used_ctr            = %10u\n"
-		"pack_report: pack_mmap_calls          = %10u\n"
-		"pack_report: pack_open_windows        = %10u / %10u\n"
-		"pack_report: pack_mapped              = "
-			"%10" SZ_FMT " / %10" SZ_FMT "\n",
-		pack_used_ctr,
-		pack_mmap_calls,
-		pack_open_windows, peak_pack_open_windows,
-		sz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));
-}
-
 /*
  * Open and mmap the index file at path, perform a couple of
  * consistency checks, then record its information to p.  Return 0 on
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#27 lore raw

PATCH v2 4/25 pack: move open_pack_index(), parse_pack_index()

In reply to Jonathan Tan

alloc_packed_git() in packfile.c is duplicated from sha1_file.c. In a subsequent commit, alloc_packed_git() will be removed from sha1_file.c.

---
 builtin/count-objects.c |   1 +
 cache.h                 |   8 ---
 pack.h                  |   8 +++
 packfile.c              | 149 ++++++++++++++++++++++++++++++++++++++++++++++++
 sha1_file.c             | 140 ---------------------------------------------
 sha1_name.c             |   1 +
 6 files changed, 159 insertions(+), 148 deletions(-)
builtin/count-objects.c +1 -0
index 1d82e61f2..185d3190a 100644
@@ -10,6 +10,7 @@
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
+#include "pack.h"
 
 static unsigned long garbage;
 static off_t size_garbage;
cache.h +0 -8
index c7f802e4a..5d6839525 100644
@@ -1603,8 +1603,6 @@ struct pack_entry {
 	struct packed_git *p;
 };
 
-extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
-
 /* A hook to report invalid files in pack directory */
 #define PACKDIR_FILE_PACK 1
 #define PACKDIR_FILE_IDX 2
@@ -1639,12 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * mmap the index file for the specified packfile (if it is not
- * already mmapped).  Return 0 on success.
- */
-extern int open_pack_index(struct packed_git *);
-
 /*
  * munmap the index file for the specified packfile (if it is
  * currently mmapped).
pack.h +8 -0
index 6098bfe40..5be0ed42a 100644
@@ -135,4 +135,12 @@ extern size_t pack_mapped;
 
 extern void pack_report(void);
 
+/*
+ * mmap the index file for the specified packfile (if it is not
+ * already mmapped).  Return 0 on success.
+ */
+extern int open_pack_index(struct packed_git *);
+
+extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
+
 #endif
packfile.c +149 -0

157 lines not on this page · raw message

sha1_file.c +0 -140

152 lines not on this page · raw message

sha1_name.c +1 -0
index 74fcb6d78..28b7c9fd8 100644
@@ -9,6 +9,7 @@
 #include "remote.h"
 #include "dir.h"
 #include "sha1-array.h"
+#include "pack.h"
 
 static int get_sha1_oneline(const char *, unsigned char *, struct commit_list *);
 
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#28 lore raw

PATCH v2 5/25 pack: move release_pack_memory()

In reply to Jonathan Tan

The function unuse_one_window() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.

---
 git-compat-util.h |  2 --
 pack.h            |  4 ++++
 packfile.c        | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 sha1_file.c       | 49 -------------------------------------------------
 4 files changed, 53 insertions(+), 51 deletions(-)
git-compat-util.h +0 -2
index db9c22de7..201056e2d 100644
@@ -749,8 +749,6 @@ const char *inet_ntop(int af, const void *src, char *dst, size_t size);
 extern int git_atexit(void (*handler)(void));
 #endif
 
-extern void release_pack_memory(size_t);
-
 typedef void (*try_to_free_t)(size_t);
 extern try_to_free_t set_try_to_free_routine(try_to_free_t);
 
pack.h +4 -0
index 5be0ed42a..c16220586 100644
@@ -143,4 +143,8 @@ extern int open_pack_index(struct packed_git *);
 
 extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
 
+extern int unuse_one_window(struct packed_git *current);
+
+extern void release_pack_memory(size_t);
+
 #endif
packfile.c +49 -0

52 lines not on this page · raw message

sha1_file.c +0 -49

55 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#29 lore raw

PATCH v2 6/25 pack: move pack-closing functions

In reply to Jonathan Tan

The function close_pack_fd() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.

---
 builtin/am.c    |  1 +
 builtin/clone.c |  1 +
 builtin/fetch.c |  1 +
 builtin/merge.c |  1 +
 cache.h         |  8 --------
 pack.h          |  9 +++++++++
 packfile.c      | 54 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 sha1_file.c     | 55 -------------------------------------------------------
 8 files changed, 67 insertions(+), 63 deletions(-)
builtin/am.c +1 -0
index c973bd96d..c38dd10a3 100644
@@ -31,6 +31,7 @@
 #include "mailinfo.h"
 #include "apply.h"
 #include "string-list.h"
+#include "pack.h"
 
 /**
  * Returns 1 if the file is empty or does not exist, 0 otherwise.
builtin/clone.c +1 -0
index 08b5cc433..53410a45d 100644
@@ -25,6 +25,7 @@
 #include "remote.h"
 #include "run-command.h"
 #include "connected.h"
+#include "pack.h"
 
 /*
  * Overall FIXMEs:
builtin/fetch.c +1 -0
index c87e59f3b..196a3bfc4 100644
@@ -17,6 +17,7 @@
 #include "connected.h"
 #include "argv-array.h"
 #include "utf8.h"
+#include "pack.h"
 
 static const char * const builtin_fetch_usage[] = {
 	N_("git fetch [<options>] [<repository> [<refspec>...]]"),
builtin/merge.c +1 -0
index 900bafdb4..9cff4b276 100644
@@ -32,6 +32,7 @@
 #include "gpg-interface.h"
 #include "sequencer.h"
 #include "string-list.h"
+#include "pack.h"
 
 #define DEFAULT_TWOHEAD (1<<0)
 #define DEFAULT_OCTOPUS (1<<1)
cache.h +0 -8
index 5d6839525..25a21a61f 100644
@@ -1637,15 +1637,7 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * munmap the index file for the specified packfile (if it is
- * currently mmapped).
- */
-extern void close_pack_index(struct packed_git *);
-
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
-extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
pack.h +9 -0
index c16220586..fd4668528 100644
@@ -147,4 +147,13 @@ extern int unuse_one_window(struct packed_git *current);
 
 extern void release_pack_memory(size_t);
 
+extern void close_pack_windows(struct packed_git *);
+extern int close_pack_fd(struct packed_git *);
+/*
+ * munmap the index file for the specified packfile (if it is
+ * currently mmapped).
+ */
+extern void close_pack_index(struct packed_git *);
+extern void close_all_packs(void);
+
 #endif
packfile.c +54 -0

57 lines not on this page · raw message

sha1_file.c +0 -55

67 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#30 lore raw

PATCH v2 7/25 pack: move use_pack()

In reply to Jonathan Tan

The function open_packed_git() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.

---
 cache.h     |   1 -
 pack.h      |  14 +--
 packfile.c  | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 sha1_file.c | 285 --------------------------------------------------------
 streaming.c |   1 +
 5 files changed, 299 insertions(+), 305 deletions(-)
cache.h +0 -1
index 25a21a61f..dd9f9a9ae 100644
@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
pack.h +4 -10
index fd4668528..bf2b99bf9 100644
@@ -124,14 +124,7 @@ extern char *sha1_pack_name(const unsigned char *sha1);
  */
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 
-extern unsigned int pack_used_ctr;
-extern unsigned int pack_mmap_calls;
-extern unsigned int peak_pack_open_windows;
-extern unsigned int pack_open_windows;
 extern unsigned int pack_open_fds;
-extern unsigned int pack_max_fds;
-extern size_t peak_pack_mapped;
-extern size_t pack_mapped;
 
 extern void pack_report(void);
 
@@ -143,12 +136,9 @@ extern int open_pack_index(struct packed_git *);
 
 extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
 
-extern int unuse_one_window(struct packed_git *current);
-
 extern void release_pack_memory(size_t);
 
 extern void close_pack_windows(struct packed_git *);
-extern int close_pack_fd(struct packed_git *);
 /*
  * munmap the index file for the specified packfile (if it is
  * currently mmapped).
@@ -156,4 +146,8 @@ extern int close_pack_fd(struct packed_git *);
 extern void close_pack_index(struct packed_git *);
 extern void close_all_packs(void);
 
+extern int open_packed_git(struct packed_git *p);
+
+extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
+
 #endif
packfile.c +294 -9

325 lines not on this page · raw message

sha1_file.c +0 -285

297 lines not on this page · raw message

streaming.c +1 -0
index 9afa66b8b..f657018cf 100644
@@ -3,6 +3,7 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "pack.h"
 
 enum input_source {
 	stream_error = -1,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#31 lore raw

PATCH v2 8/25 pack: move unuse_pack()

In reply to Jonathan Tan
---
 cache.h     | 1 -
 pack.h      | 1 +
 packfile.c  | 9 +++++++++
 sha1_file.c | 9 ---------
 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -1
index dd9f9a9ae..4812f3a63 100644
@@ -1637,7 +1637,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
pack.h +1 -0
index bf2b99bf9..3876e9ae6 100644
@@ -149,5 +149,6 @@ extern void close_all_packs(void);
 extern int open_packed_git(struct packed_git *p);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
+extern void unuse_pack(struct pack_window **);
 
 #endif
packfile.c +9 -0
index 85cb65558..93526ea7b 100644
@@ -596,3 +596,12 @@ unsigned char *use_pack(struct packed_git *p,
 		*left = win->len - xsize_t(offset);
 	return win->base + offset;
 }
+
+void unuse_pack(struct pack_window **w_cursor)
+{
+	struct pack_window *w = *w_cursor;
+	if (w) {
+		w->inuse_cnt--;
+		*w_cursor = NULL;
+	}
+}
sha1_file.c +0 -9
index 8f17a07e9..12501ef06 100644
@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
-void unuse_pack(struct pack_window **w_cursor)
-{
-	struct pack_window *w = *w_cursor;
-	if (w) {
-		w->inuse_cnt--;
-		*w_cursor = NULL;
-	}
-}
-
 static struct packed_git *alloc_packed_git(int extra)
 {
 	struct packed_git *p = xmalloc(st_add(sizeof(*p), extra));
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#32 lore raw

PATCH v2 9/25 pack: move add_packed_git()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 connected.c |  1 +
 pack.h      |  1 +
 packfile.c  | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 sha1_file.c | 61 -------------------------------------------------------------
 5 files changed, 55 insertions(+), 62 deletions(-)
cache.h +0 -1
index 4812f3a63..bf93477e8 100644
@@ -1638,7 +1638,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
 extern int odb_pack_keep(const char *name);
 
 extern void clear_delta_base_cache(void);
-extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
 /*
  * Make sure that a pointer access into an mmap'd index file is within bounds,
connected.c +1 -0
index 136c2ac16..3e3f0148c 100644
@@ -3,6 +3,7 @@
 #include "sigchain.h"
 #include "connected.h"
 #include "transport.h"
+#include "pack.h"
 
 /*
  * If we feed all the commits we want to verify to this command
pack.h +1 -0
index 3876e9ae6..c1f3ff32d 100644
@@ -150,5 +150,6 @@ extern int open_packed_git(struct packed_git *p);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void unuse_pack(struct pack_window **);
+extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
 #endif
packfile.c +53 -0

56 lines not on this page · raw message

sha1_file.c +0 -61

67 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#33 lore raw

PATCH v2 10/25 pack: move install_packed_git()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 pack.h      |  4 ++--
 packfile.c  | 11 ++++++++++-
 sha1_file.c |  9 ---------
 4 files changed, 12 insertions(+), 13 deletions(-)
cache.h +0 -1
index bf93477e8..41562dc0b 100644
@@ -1611,7 +1611,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
 extern void prepare_packed_git(void);
 extern void reprepare_packed_git(void);
-extern void install_packed_git(struct packed_git *pack);
 
 /*
  * Give a rough count of objects in the repository. This sacrifices accuracy
pack.h +2 -2
index c1f3ff32d..576c4fc7c 100644
@@ -124,8 +124,6 @@ extern char *sha1_pack_name(const unsigned char *sha1);
  */
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 
-extern unsigned int pack_open_fds;
-
 extern void pack_report(void);
 
 /*
@@ -152,4 +150,6 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t
 extern void unuse_pack(struct pack_window **);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
+extern void install_packed_git(struct packed_git *pack);
+
 #endif
packfile.c +10 -1
index efe0ed3e8..4eb65e460 100644
@@ -28,7 +28,7 @@ static unsigned int pack_used_ctr;
 static unsigned int pack_mmap_calls;
 static unsigned int peak_pack_open_windows;
 static unsigned int pack_open_windows;
-unsigned int pack_open_fds;
+static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
@@ -658,3 +658,12 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
 		hashclr(p->sha1);
 	return p;
 }
+
+void install_packed_git(struct packed_git *pack)
+{
+	if (pack->pack_fd != -1)
+		pack_open_fds++;
+
+	pack->next = packed_git;
+	packed_git = pack;
+}
sha1_file.c +0 -9
index 7f12b1ee0..b956ca0c9 100644
@@ -717,15 +717,6 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
-void install_packed_git(struct packed_git *pack)
-{
-	if (pack->pack_fd != -1)
-		pack_open_fds++;
-
-	pack->next = packed_git;
-	packed_git = pack;
-}
-
 void (*report_garbage)(unsigned seen_bits, const char *path);
 
 static void report_helper(const struct string_list *list,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#34 lore raw

PATCH v2 11/25 pack: move {,re}prepare_packed_git and approximate_object_count

In reply to Jonathan Tan
---
 builtin/gc.c   |   1 +
 cache.h        |  15 ----
 http-backend.c |   1 +
 pack.h         |  15 ++++
 packfile.c     | 216 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 path.c         |   1 +
 server-info.c  |   1 +
 sha1_file.c    | 214 --------------------------------------------------------
 8 files changed, 235 insertions(+), 229 deletions(-)
builtin/gc.c +1 -0
index e6b84475a..f4fe023d3 100644
@@ -19,6 +19,7 @@
 #include "sigchain.h"
 #include "argv-array.h"
 #include "commit.h"
+#include "pack.h"
 
 #define FAILED_RUN "failed to run %s"
 
cache.h +0 -15
index 41562dc0b..f020dfade 100644
@@ -1603,21 +1603,6 @@ struct pack_entry {
 	struct packed_git *p;
 };
 
-/* A hook to report invalid files in pack directory */
-#define PACKDIR_FILE_PACK 1
-#define PACKDIR_FILE_IDX 2
-#define PACKDIR_FILE_GARBAGE 4
-extern void (*report_garbage)(unsigned seen_bits, const char *path);
-
-extern void prepare_packed_git(void);
-extern void reprepare_packed_git(void);
-
-/*
- * Give a rough count of objects in the repository. This sacrifices accuracy
- * for speed.
- */
-unsigned long approximate_object_count(void);
-
 extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
 					 struct packed_git *packs);
 
http-backend.c +1 -0
index 519025d2c..12f7d421f 100644
@@ -9,6 +9,7 @@
 #include "string-list.h"
 #include "url.h"
 #include "argv-array.h"
+#include "pack.h"
 
 static const char content_type[] = "Content-Type";
 static const char content_length[] = "Content-Length";
pack.h +15 -0
index 576c4fc7c..cad5ed488 100644
@@ -152,4 +152,19 @@ extern struct packed_git *add_packed_git(const char *path, size_t path_len, int
 
 extern void install_packed_git(struct packed_git *pack);
 
+/* A hook to report invalid files in pack directory */
+#define PACKDIR_FILE_PACK 1
+#define PACKDIR_FILE_IDX 2
+#define PACKDIR_FILE_GARBAGE 4
+extern void (*report_garbage)(unsigned seen_bits, const char *path);
+
+extern void prepare_packed_git(void);
+extern void reprepare_packed_git(void);
+
+/*
+ * Give a rough count of objects in the repository. This sacrifices accuracy
+ * for speed.
+ */
+unsigned long approximate_object_count(void);
+
 #endif
packfile.c +216 -0

225 lines not on this page · raw message

path.c +1 -0
index e485f9f93..ae3f2f65f 100644
@@ -9,6 +9,7 @@
 #include "worktree.h"
 #include "submodule-config.h"
 #include "path.h"
+#include "pack.h"
 
 static int get_st_mode_bits(const char *path, int *mode)
 {
server-info.c +1 -0
index 5ec5b1d82..e9bc18a8c 100644
@@ -3,6 +3,7 @@
 #include "object.h"
 #include "commit.h"
 #include "tag.h"
+#include "pack.h"
 
 /*
  * Create the file "path" by writing to a temporary file and renaming
sha1_file.c +0 -214

220 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#35 lore raw

PATCH v2 12/25 pack: move unpack_object_header()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 pack.h      |  2 ++
 packfile.c  | 25 +++++++++++++++++++++++++
 sha1_file.c | 25 -------------------------
 4 files changed, 27 insertions(+), 26 deletions(-)
cache.h +0 -1
index f020dfade..9c70759a6 100644
@@ -1661,7 +1661,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)
 
 extern int is_pack_valid(struct packed_git *);
 extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
-extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
 extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
 extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
 
pack.h +2 -0
index cad5ed488..4a7f88a38 100644
@@ -167,4 +167,6 @@ extern void reprepare_packed_git(void);
  */
 unsigned long approximate_object_count(void);
 
+extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
+
 #endif
packfile.c +25 -0
index a517172f7..6e4f1c6e3 100644
@@ -883,3 +883,28 @@ void reprepare_packed_git(void)
 	prepare_packed_git_run_once = 0;
 	prepare_packed_git();
 }
+
+unsigned long unpack_object_header_buffer(const unsigned char *buf,
+		unsigned long len, enum object_type *type, unsigned long *sizep)
+{
+	unsigned shift;
+	unsigned long size, c;
+	unsigned long used = 0;
+
+	c = buf[used++];
+	*type = (c >> 4) & 7;
+	size = c & 15;
+	shift = 4;
+	while (c & 0x80) {
+		if (len <= used || bitsizeof(long) <= shift) {
+			error("bad object header");
+			size = used = 0;
+			break;
+		}
+		c = buf[used++];
+		size += (c & 0x7f) << shift;
+		shift += 7;
+	}
+	*sizep = size;
+	return used;
+}
sha1_file.c +0 -25
index bbce60f1c..1f4b4ba2c 100644
@@ -913,31 +913,6 @@ void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 	return map_sha1_file_1(NULL, sha1, size);
 }
 
-unsigned long unpack_object_header_buffer(const unsigned char *buf,
-		unsigned long len, enum object_type *type, unsigned long *sizep)
-{
-	unsigned shift;
-	unsigned long size, c;
-	unsigned long used = 0;
-
-	c = buf[used++];
-	*type = (c >> 4) & 7;
-	size = c & 15;
-	shift = 4;
-	while (c & 0x80) {
-		if (len <= used || bitsizeof(long) <= shift) {
-			error("bad object header");
-			size = used = 0;
-			break;
-		}
-		c = buf[used++];
-		size += (c & 0x7f) << shift;
-		shift += 7;
-	}
-	*sizep = size;
-	return used;
-}
-
 static int unpack_sha1_short_header(git_zstream *stream,
 				    unsigned char *map, unsigned long mapsize,
 				    void *buffer, unsigned long bufsiz)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#36 lore raw

PATCH v2 13/25 pack: move get_size_from_delta()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 pack.h      |  1 +
 packfile.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 sha1_file.c | 39 ---------------------------------------
 4 files changed, 41 insertions(+), 40 deletions(-)
cache.h +0 -1
index 9c70759a6..e29918c75 100644
@@ -1661,7 +1661,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)
 
 extern int is_pack_valid(struct packed_git *);
 extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
-extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
 extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
 
 /*
pack.h +1 -0
index 4a7f88a38..69c92d8d2 100644
@@ -168,5 +168,6 @@ extern void reprepare_packed_git(void);
 unsigned long approximate_object_count(void);
 
 extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
+extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
 
 #endif
packfile.c +40 -0
index 6e4f1c6e3..511afad85 100644
@@ -3,6 +3,7 @@
 #include "pack.h"
 #include "dir.h"
 #include "mergesort.h"
+#include "delta.h"
 
 char *odb_pack_name(struct strbuf *buf,
 		    const unsigned char *sha1,
@@ -908,3 +909,42 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,
 	*sizep = size;
 	return used;
 }
+
+unsigned long get_size_from_delta(struct packed_git *p,
+				  struct pack_window **w_curs,
+			          off_t curpos)
+{
+	const unsigned char *data;
+	unsigned char delta_head[20], *in;
+	git_zstream stream;
+	int st;
+
+	memset(&stream, 0, sizeof(stream));
+	stream.next_out = delta_head;
+	stream.avail_out = sizeof(delta_head);
+
+	git_inflate_init(&stream);
+	do {
+		in = use_pack(p, w_curs, curpos, &stream.avail_in);
+		stream.next_in = in;
+		st = git_inflate(&stream, Z_FINISH);
+		curpos += stream.next_in - in;
+	} while ((st == Z_OK || st == Z_BUF_ERROR) &&
+		 stream.total_out < sizeof(delta_head));
+	git_inflate_end(&stream);
+	if ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head)) {
+		error("delta data unpack-initial failed");
+		return 0;
+	}
+
+	/* Examine the initial part of the delta to figure out
+	 * the result size.
+	 */
+	data = delta_head;
+
+	/* ignore base size */
+	get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
+
+	/* Read the result size */
+	return get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
+}
sha1_file.c +0 -39
index 1f4b4ba2c..7d354d9b6 100644
@@ -1099,45 +1099,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-unsigned long get_size_from_delta(struct packed_git *p,
-				  struct pack_window **w_curs,
-			          off_t curpos)
-{
-	const unsigned char *data;
-	unsigned char delta_head[20], *in;
-	git_zstream stream;
-	int st;
-
-	memset(&stream, 0, sizeof(stream));
-	stream.next_out = delta_head;
-	stream.avail_out = sizeof(delta_head);
-
-	git_inflate_init(&stream);
-	do {
-		in = use_pack(p, w_curs, curpos, &stream.avail_in);
-		stream.next_in = in;
-		st = git_inflate(&stream, Z_FINISH);
-		curpos += stream.next_in - in;
-	} while ((st == Z_OK || st == Z_BUF_ERROR) &&
-		 stream.total_out < sizeof(delta_head));
-	git_inflate_end(&stream);
-	if ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head)) {
-		error("delta data unpack-initial failed");
-		return 0;
-	}
-
-	/* Examine the initial part of the delta to figure out
-	 * the result size.
-	 */
-	data = delta_head;
-
-	/* ignore base size */
-	get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
-
-	/* Read the result size */
-	return get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
-}
-
 static off_t get_delta_base(struct packed_git *p,
 				    struct pack_window **w_curs,
 				    off_t *curpos,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#37 lore raw

PATCH v2 14/25 pack: move unpack_object_header()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 pack.h      |  1 +
 packfile.c  | 26 ++++++++++++++++++++++++++
 sha1_file.c | 26 --------------------------
 4 files changed, 27 insertions(+), 27 deletions(-)
cache.h +0 -1
index e29918c75..1ae3333a0 100644
@@ -1661,7 +1661,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)
 
 extern int is_pack_valid(struct packed_git *);
 extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
-extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
 
 /*
  * Iterate over the files in the loose-object parts of the object
pack.h +1 -0
index 69c92d8d2..5e3552392 100644
@@ -169,5 +169,6 @@ unsigned long approximate_object_count(void);
 
 extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
 extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
+extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
 
 #endif
packfile.c +26 -0
index 511afad85..a4db78ea0 100644
@@ -948,3 +948,29 @@ unsigned long get_size_from_delta(struct packed_git *p,
 	/* Read the result size */
 	return get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
 }
+
+int unpack_object_header(struct packed_git *p,
+			 struct pack_window **w_curs,
+			 off_t *curpos,
+			 unsigned long *sizep)
+{
+	unsigned char *base;
+	unsigned long left;
+	unsigned long used;
+	enum object_type type;
+
+	/* use_pack() assures us we have [base, base + 20) available
+	 * as a range that we can look at.  (Its actually the hash
+	 * size that is assured.)  With our object header encoding
+	 * the maximum deflated object size is 2^137, which is just
+	 * insane, so we know won't exceed what we have been given.
+	 */
+	base = use_pack(p, w_curs, *curpos, &left);
+	used = unpack_object_header_buffer(base, left, &type, sizep);
+	if (!used) {
+		type = OBJ_BAD;
+	} else
+		*curpos += used;
+
+	return type;
+}
sha1_file.c +0 -26
index 7d354d9b6..f3bcdae17 100644
@@ -1170,32 +1170,6 @@ static const unsigned char *get_delta_base_sha1(struct packed_git *p,
 		return NULL;
 }
 
-int unpack_object_header(struct packed_git *p,
-			 struct pack_window **w_curs,
-			 off_t *curpos,
-			 unsigned long *sizep)
-{
-	unsigned char *base;
-	unsigned long left;
-	unsigned long used;
-	enum object_type type;
-
-	/* use_pack() assures us we have [base, base + 20) available
-	 * as a range that we can look at.  (Its actually the hash
-	 * size that is assured.)  With our object header encoding
-	 * the maximum deflated object size is 2^137, which is just
-	 * insane, so we know won't exceed what we have been given.
-	 */
-	base = use_pack(p, w_curs, *curpos, &left);
-	used = unpack_object_header_buffer(base, left, &type, sizep);
-	if (!used) {
-		type = OBJ_BAD;
-	} else
-		*curpos += used;
-
-	return type;
-}
-
 static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)
 {
 	int type;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#38 lore raw

PATCH v2 15/25 sha1_file: set whence in storage-specific info fn

In reply to Jonathan Tan

Move the setting of oi->whence to sha1_loose_object_info() and packed_object_info(). This allows sha1_object_info_extended() to not need to know about the delta base cache.

---
 sha1_file.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
sha1_file.c +6 -7
index f3bcdae17..9eadda388 100644
@@ -1473,6 +1473,9 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,
 			hashclr(oi->delta_base_sha1);
 	}
 
+	oi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :
+							  OI_PACKED;
+
 out:
 	unuse_pack(&w_curs);
 	return type;
@@ -2002,6 +2005,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (oi->sizep == &size_scratch)
 		oi->sizep = NULL;
 	strbuf_release(&hdrbuf);
+	oi->whence = OI_LOOSE;
 	return (status < 0) ? status : 0;
 }
 
@@ -2039,10 +2043,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 
 	if (!find_pack_entry(real, &e)) {
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags)) {
-			oi->whence = OI_LOOSE;
+		if (!sha1_loose_object_info(real, oi, flags))
 			return 0;
-		}
 
 		/* Not a loose object; someone else may have just packed it. */
 		if (flags & OBJECT_INFO_QUICK) {
@@ -2065,10 +2067,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 	if (rtype < 0) {
 		mark_bad_packed_object(e.p, real);
 		return sha1_object_info_extended(real, oi, 0);
-	} else if (in_delta_base_cache(e.p, e.offset)) {
-		oi->whence = OI_DBCACHED;
-	} else {
-		oi->whence = OI_PACKED;
+	} else if (oi->whence == OI_PACKED) {
 		oi->u.packed.offset = e.offset;
 		oi->u.packed.pack = e.p;
 		oi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#39 lore raw

PATCH v2 16/25 sha1_file: remove read_packed_sha1()

In reply to Jonathan Tan

Use read_object() in its place instead. This avoids duplication of code.

This makes force_object_loose() slightly slower (because of a redundant check of loose object storage), but only in the error case.

---
 sha1_file.c | 26 +-------------------------
 1 file changed, 1 insertion(+), 25 deletions(-)
sha1_file.c +1 -25
index 9eadda388..9e5444334 100644
@@ -2091,30 +2091,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)
 	return type;
 }
 
-static void *read_packed_sha1(const unsigned char *sha1,
-			      enum object_type *type, unsigned long *size)
-{
-	struct pack_entry e;
-	void *data;
-
-	if (!find_pack_entry(sha1, &e))
-		return NULL;
-	data = cache_or_unpack_entry(e.p, e.offset, size, type);
-	if (!data) {
-		/*
-		 * We're probably in deep shit, but let's try to fetch
-		 * the required object anyway from another pack or loose.
-		 * This should happen only in the presence of a corrupted
-		 * pack, and is better than failing outright.
-		 */
-		error("failed to read object %s at offset %"PRIuMAX" from %s",
-		      sha1_to_hex(sha1), (uintmax_t)e.offset, e.p->pack_name);
-		mark_bad_packed_object(e.p, sha1);
-		data = read_object(sha1, type, size);
-	}
-	return data;
-}
-
 int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,
 		      unsigned char *sha1)
 {
@@ -2497,7 +2473,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)
 
 	if (has_loose_object(sha1))
 		return 0;
-	buf = read_packed_sha1(sha1, &type, &len);
+	buf = read_object(sha1, &type, &len);
 	if (!buf)
 		return error("cannot read sha1_file for %s", sha1_to_hex(sha1));
 	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %lu", typename(type), len) + 1;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#40 lore raw

PATCH v2 17/25 pack: move packed_object_info(), unpack_entry()

In reply to Jonathan Tan

Both sha1_file.c and packfile.c now need read_object(), so a copy of read_object() was created in packfile.c.

This patch makes both mark_bad_packed_object() and has_packed_and_bad() global. Unlike most of the other patches in this series, these 2 functions need to remain global.

---
 cache.h     |   7 -
 pack.h      |  11 +
 packfile.c  | 660 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 sha1_file.c | 676 ++----------------------------------------------------------
 4 files changed, 685 insertions(+), 669 deletions(-)
cache.h +0 -7
index 1ae3333a0..b14098bf1 100644
@@ -1186,9 +1186,6 @@ extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
-/* global flag to enable extra checks when accessing packed objects */
-extern int do_check_packed_object_crc;
-
 extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);
 
 extern int finalize_object_file(const char *tmpfile, const char *filename);
@@ -1621,8 +1618,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-extern void clear_delta_base_cache(void);
-
 /*
  * Make sure that a pointer access into an mmap'd index file is within bounds,
  * and can provide at least 8 bytes of data.
@@ -1660,7 +1655,6 @@ extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
 extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);
 
 extern int is_pack_valid(struct packed_git *);
-extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
 
 /*
  * Iterate over the files in the loose-object parts of the object
@@ -1771,7 +1765,6 @@ struct object_info {
 /* Do not retry packed storage after checking packed and loose storage */
 #define OBJECT_INFO_QUICK 8
 extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);
-extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);
 
 /* Dumb servers support */
 extern int update_server_info(int);
pack.h +11 -0
index 5e3552392..2e6f357c3 100644
@@ -171,4 +171,15 @@ extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsig
 extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
 extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
 
+extern void clear_delta_base_cache(void);
+
+/* global flag to enable extra checks when accessing packed objects */
+extern int do_check_packed_object_crc;
+
+extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);
+extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
+
+extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);
+extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);
+
 #endif
packfile.c +660 -0

669 lines not on this page · raw message

sha1_file.c +14 -662

700 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#41 lore raw

PATCH v2 18/25 pack: move nth_packed_object_{sha1,oid}

In reply to Jonathan Tan
---
 cache.h     | 14 --------------
 pack.h      | 14 ++++++++++++++
 packfile.c  | 31 +++++++++++++++++++++++++++++++
 sha1_file.c | 31 -------------------------------
 4 files changed, 45 insertions(+), 45 deletions(-)
cache.h +0 -14
index b14098bf1..f083d532e 100644
@@ -1628,20 +1628,6 @@ extern int odb_pack_keep(const char *name);
  */
 extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);
 
-/*
- * Return the SHA-1 of the nth object within the specified packfile.
- * Open the index if it is not already open.  The return value points
- * at the SHA-1 within the mmapped index.  Return NULL if there is an
- * error.
- */
-extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n);
-/*
- * Like nth_packed_object_sha1, but write the data into the object specified by
- * the the first argument.  Returns the first argument on success, and NULL on
- * error.
- */
-extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);
-
 /*
  * Return the offset of the nth object within the specified packfile.
  * The index must already be opened.
pack.h +14 -0
index 2e6f357c3..023c97b37 100644
@@ -182,4 +182,18 @@ extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsign
 extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);
 extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);
 
+/*
+ * Return the SHA-1 of the nth object within the specified packfile.
+ * Open the index if it is not already open.  The return value points
+ * at the SHA-1 within the mmapped index.  Return NULL if there is an
+ * error.
+ */
+extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n);
+/*
+ * Like nth_packed_object_sha1, but write the data into the object specified by
+ * the the first argument.  Returns the first argument on success, and NULL on
+ * error.
+ */
+extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);
+
 #endif
packfile.c +31 -0
index a3745f9df..b16cf648a 100644
@@ -1634,3 +1634,34 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,
 
 	return data;
 }
+
+const unsigned char *nth_packed_object_sha1(struct packed_git *p,
+					    uint32_t n)
+{
+	const unsigned char *index = p->index_data;
+	if (!index) {
+		if (open_pack_index(p))
+			return NULL;
+		index = p->index_data;
+	}
+	if (n >= p->num_objects)
+		return NULL;
+	index += 4 * 256;
+	if (p->index_version == 1) {
+		return index + 24 * n + 4;
+	} else {
+		index += 8;
+		return index + 20 * n;
+	}
+}
+
+const struct object_id *nth_packed_object_oid(struct object_id *oid,
+					      struct packed_git *p,
+					      uint32_t n)
+{
+	const unsigned char *hash = nth_packed_object_sha1(p, n);
+	if (!hash)
+		return NULL;
+	hashcpy(oid->hash, hash);
+	return oid;
+}
sha1_file.c +0 -31
index fe7e0db76..4cd2b1809 100644
@@ -1073,37 +1073,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-const unsigned char *nth_packed_object_sha1(struct packed_git *p,
-					    uint32_t n)
-{
-	const unsigned char *index = p->index_data;
-	if (!index) {
-		if (open_pack_index(p))
-			return NULL;
-		index = p->index_data;
-	}
-	if (n >= p->num_objects)
-		return NULL;
-	index += 4 * 256;
-	if (p->index_version == 1) {
-		return index + 24 * n + 4;
-	} else {
-		index += 8;
-		return index + 20 * n;
-	}
-}
-
-const struct object_id *nth_packed_object_oid(struct object_id *oid,
-					      struct packed_git *p,
-					      uint32_t n)
-{
-	const unsigned char *hash = nth_packed_object_sha1(p, n);
-	if (!hash)
-		return NULL;
-	hashcpy(oid->hash, hash);
-	return oid;
-}
-
 void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
 {
 	const unsigned char *ptr = vptr;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#42 lore raw

PATCH v2 19/25 pack: move check_pack_index_ptr(), nth_packed_object_offset()

In reply to Jonathan Tan
---
 cache.h     | 16 ----------------
 pack.h      | 16 ++++++++++++++++
 packfile.c  | 33 +++++++++++++++++++++++++++++++++
 sha1_file.c | 33 ---------------------------------
 4 files changed, 49 insertions(+), 49 deletions(-)
cache.h +0 -16
index f083d532e..7686ccb30 100644
@@ -1618,22 +1618,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * Make sure that a pointer access into an mmap'd index file is within bounds,
- * and can provide at least 8 bytes of data.
- *
- * Note that this is only necessary for variable-length segments of the file
- * (like the 64-bit extended offset table), as we compare the size to the
- * fixed-length parts when we open the file.
- */
-extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);
-
-/*
- * Return the offset of the nth object within the specified packfile.
- * The index must already be opened.
- */
-extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
-
 /*
  * If the object named sha1 is present in the specified packfile,
  * return its offset within the packfile; otherwise, return 0.
pack.h +16 -0
index 023c97b37..e0e206e3c 100644
@@ -196,4 +196,20 @@ extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t
  */
 extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);
 
+/*
+ * Make sure that a pointer access into an mmap'd index file is within bounds,
+ * and can provide at least 8 bytes of data.
+ *
+ * Note that this is only necessary for variable-length segments of the file
+ * (like the 64-bit extended offset table), as we compare the size to the
+ * fixed-length parts when we open the file.
+ */
+extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);
+
+/*
+ * Return the offset of the nth object within the specified packfile.
+ * The index must already be opened.
+ */
+extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
+
 #endif
packfile.c +33 -0
index b16cf648a..94c8af991 100644
@@ -1665,3 +1665,36 @@ const struct object_id *nth_packed_object_oid(struct object_id *oid,
 	hashcpy(oid->hash, hash);
 	return oid;
 }
+
+void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
+{
+	const unsigned char *ptr = vptr;
+	const unsigned char *start = p->index_data;
+	const unsigned char *end = start + p->index_size;
+	if (ptr < start)
+		die(_("offset before start of pack index for %s (corrupt index?)"),
+		    p->pack_name);
+	/* No need to check for underflow; .idx files must be at least 8 bytes */
+	if (ptr >= end - 8)
+		die(_("offset beyond end of pack index for %s (truncated index?)"),
+		    p->pack_name);
+}
+
+off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
+{
+	const unsigned char *index = p->index_data;
+	index += 4 * 256;
+	if (p->index_version == 1) {
+		return ntohl(*((uint32_t *)(index + 24 * n)));
+	} else {
+		uint32_t off;
+		index += 8 + p->num_objects * (20 + 4);
+		off = ntohl(*((uint32_t *)(index + 4 * n)));
+		if (!(off & 0x80000000))
+			return off;
+		index += p->num_objects * 4 + (off & 0x7fffffff) * 8;
+		check_pack_index_ptr(p, index);
+		return (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |
+				   ntohl(*((uint32_t *)(index + 4)));
+	}
+}
sha1_file.c +0 -33
index 4cd2b1809..0f4d68c5a 100644
@@ -1073,39 +1073,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
-{
-	const unsigned char *ptr = vptr;
-	const unsigned char *start = p->index_data;
-	const unsigned char *end = start + p->index_size;
-	if (ptr < start)
-		die(_("offset before start of pack index for %s (corrupt index?)"),
-		    p->pack_name);
-	/* No need to check for underflow; .idx files must be at least 8 bytes */
-	if (ptr >= end - 8)
-		die(_("offset beyond end of pack index for %s (truncated index?)"),
-		    p->pack_name);
-}
-
-off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
-{
-	const unsigned char *index = p->index_data;
-	index += 4 * 256;
-	if (p->index_version == 1) {
-		return ntohl(*((uint32_t *)(index + 24 * n)));
-	} else {
-		uint32_t off;
-		index += 8 + p->num_objects * (20 + 4);
-		off = ntohl(*((uint32_t *)(index + 4 * n)));
-		if (!(off & 0x80000000))
-			return off;
-		index += p->num_objects * 4 + (off & 0x7fffffff) * 8;
-		check_pack_index_ptr(p, index);
-		return (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |
-				   ntohl(*((uint32_t *)(index + 4)));
-	}
-}
-
 off_t find_pack_entry_one(const unsigned char *sha1,
 				  struct packed_git *p)
 {
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#43 lore raw

PATCH v2 20/25 pack: move find_pack_entry_one(), is_pack_valid()

In reply to Jonathan Tan
---
 cache.h     |  8 ------
 pack.h      | 10 ++++++--
 packfile.c  | 85 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 sha1_file.c | 84 ------------------------------------------------------------
 4 files changed, 93 insertions(+), 94 deletions(-)
cache.h +0 -8
index 7686ccb30..b944aca69 100644
@@ -1618,14 +1618,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * If the object named sha1 is present in the specified packfile,
- * return its offset within the packfile; otherwise, return 0.
- */
-extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);
-
-extern int is_pack_valid(struct packed_git *);
-
 /*
  * Iterate over the files in the loose-object parts of the object
  * directory "path", triggering the following callbacks:
pack.h +8 -2
index e0e206e3c..f5bd94813 100644
@@ -144,8 +144,6 @@ extern void close_pack_windows(struct packed_git *);
 extern void close_pack_index(struct packed_git *);
 extern void close_all_packs(void);
 
-extern int open_packed_git(struct packed_git *p);
-
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void unuse_pack(struct pack_window **);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
@@ -212,4 +210,12 @@ extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);
  */
 extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
 
+/*
+ * If the object named sha1 is present in the specified packfile,
+ * return its offset within the packfile; otherwise, return 0.
+ */
+extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);
+
+extern int is_pack_valid(struct packed_git *);
+
 #endif
packfile.c +85 -0

94 lines not on this page · raw message

sha1_file.c +0 -84

90 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#44 lore raw

PATCH v2 21/25 pack: move find_sha1_pack()

In reply to Jonathan Tan
---
 cache.h       |  3 ---
 http-push.c   |  1 +
 http-walker.c |  1 +
 pack.h        |  3 +++
 packfile.c    | 13 +++++++++++++
 sha1_file.c   | 13 -------------
 6 files changed, 18 insertions(+), 16 deletions(-)
cache.h +0 -3
index b944aca69..06a8caae6 100644
@@ -1600,9 +1600,6 @@ struct pack_entry {
 	struct packed_git *p;
 };
 
-extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
-					 struct packed_git *packs);
-
 /*
  * Create a temporary file rooted in the object database directory, or
  * die on failure. The filename is taken from "pattern", which should have the
http-push.c +1 -0
index c91f40a61..4e8a227d1 100644
@@ -11,6 +11,7 @@
 #include "list-objects.h"
 #include "sigchain.h"
 #include "argv-array.h"
+#include "pack.h"
 
 #ifdef EXPAT_NEEDS_XMLPARSE_H
 #include <xmlparse.h>
http-walker.c +1 -0
index ee049cb13..d6f0af944 100644
@@ -4,6 +4,7 @@
 #include "http.h"
 #include "list.h"
 #include "transport.h"
+#include "pack.h"
 
 struct alt_base {
 	char *base;
pack.h +3 -0
index f5bd94813..0517d6542 100644
@@ -218,4 +218,7 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)
 
 extern int is_pack_valid(struct packed_git *);
 
+extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
+					 struct packed_git *packs);
+
 #endif
packfile.c +13 -0
index 71017d2ec..f16b56262 100644
@@ -1783,3 +1783,16 @@ int is_pack_valid(struct packed_git *p)
 	/* Force the pack to open to prove its valid. */
 	return !open_packed_git(p);
 }
+
+struct packed_git *find_sha1_pack(const unsigned char *sha1,
+				  struct packed_git *packs)
+{
+	struct packed_git *p;
+
+	for (p = packs; p; p = p->next) {
+		if (find_pack_entry_one(sha1, p))
+			return p;
+	}
+	return NULL;
+
+}
sha1_file.c +0 -13
index 75b9ceb39..229358663 100644
@@ -1126,19 +1126,6 @@ static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	return 0;
 }
 
-struct packed_git *find_sha1_pack(const unsigned char *sha1,
-				  struct packed_git *packs)
-{
-	struct packed_git *p;
-
-	for (p = packs; p; p = p->next) {
-		if (find_pack_entry_one(sha1, p))
-			return p;
-	}
-	return NULL;
-
-}
-
 static int sha1_loose_object_info(const unsigned char *sha1,
 				  struct object_info *oi,
 				  int flags)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#45 lore raw

PATCH v2 22/25 pack: move find_pack_entry() and make it global

In reply to Jonathan Tan

This function needs to be global as it is used by sha1_file.c and will be used by packfile.c.

---
 pack.h      |  2 ++
 packfile.c  | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 sha1_file.c | 53 -----------------------------------------------------
 3 files changed, 55 insertions(+), 53 deletions(-)
pack.h +2 -0
index 0517d6542..1021a781c 100644
@@ -221,4 +221,6 @@ extern int is_pack_valid(struct packed_git *);
 extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
 					 struct packed_git *packs);
 
+extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);
+
 #endif
packfile.c +53 -0

56 lines not on this page · raw message

sha1_file.c +0 -53

59 lines not on this page · raw message

Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#46 lore raw

PATCH v2 23/25 pack: move has_sha1_pack()

In reply to Jonathan Tan
---
 builtin/prune-packed.c | 1 +
 cache.h                | 2 --
 diff.c                 | 1 +
 pack.h                 | 2 ++
 packfile.c             | 6 ++++++
 revision.c             | 1 +
 sha1_file.c            | 6 ------
 7 files changed, 11 insertions(+), 8 deletions(-)
builtin/prune-packed.c +1 -0
index ac978ad40..79130aa2e 100644
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "progress.h"
 #include "parse-options.h"
+#include "pack.h"
 
 static const char * const prune_packed_usage[] = {
 	N_("git prune-packed [-n | --dry-run] [-q | --quiet]"),
cache.h +0 -2
index 06a8caae6..d96d36d50 100644
@@ -1190,8 +1190,6 @@ extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned l
 
 extern int finalize_object_file(const char *tmpfile, const char *filename);
 
-extern int has_sha1_pack(const unsigned char *sha1);
-
 /*
  * Open the loose object at path, check its sha1, and return the contents,
  * type, and size. If the object is a blob, then "contents" may return NULL,
diff.c +1 -0
index 85e714f6c..6bbc46326 100644
@@ -20,6 +20,7 @@
 #include "string-list.h"
 #include "argv-array.h"
 #include "graph.h"
+#include "pack.h"
 
 #ifdef NO_FAST_WORKING_DIRECTORY
 #define FAST_WORKING_DIRECTORY 0
pack.h +2 -0
index 1021a781c..ce0e15deb 100644
@@ -223,4 +223,6 @@ extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
 
 extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);
 
+extern int has_sha1_pack(const unsigned char *sha1);
+
 #endif
packfile.c +6 -0
index 0f1e3338b..507f65236 100644
@@ -1849,3 +1849,9 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	}
 	return 0;
 }
+
+int has_sha1_pack(const unsigned char *sha1)
+{
+	struct pack_entry e;
+	return find_pack_entry(sha1, &e);
+}
revision.c +1 -0
index 6603af944..2868c4fc8 100644
@@ -19,6 +19,7 @@
 #include "dir.h"
 #include "cache-tree.h"
 #include "bisect.h"
+#include "pack.h"
 
 volatile show_early_output_fn_t show_early_output;
 
sha1_file.c +0 -6
index 1a505eae5..2610ea057 100644
@@ -1629,12 +1629,6 @@ int has_pack_index(const unsigned char *sha1)
 	return 1;
 }
 
-int has_sha1_pack(const unsigned char *sha1)
-{
-	struct pack_entry e;
-	return find_pack_entry(sha1, &e);
-}
-
 int has_sha1_file_with_flags(const unsigned char *sha1, int flags)
 {
 	if (!startup_info->have_repository)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#47 lore raw

PATCH v2 24/25 pack: move has_pack_index()

In reply to Jonathan Tan
---
 cache.h     | 2 --
 pack.h      | 2 ++
 packfile.c  | 8 ++++++++
 sha1_file.c | 8 --------
 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -2
index d96d36d50..656b39d51 100644
@@ -1225,8 +1225,6 @@ extern int has_object_file_with_flags(const struct object_id *oid, int flags);
  */
 extern int has_loose_object_nonlocal(const unsigned char *sha1);
 
-extern int has_pack_index(const unsigned char *sha1);
-
 extern void assert_sha1_type(const unsigned char *sha1, enum object_type expect);
 
 /* Helper to check and "touch" a file */
pack.h +2 -0
index ce0e15deb..2c2a347ba 100644
@@ -225,4 +225,6 @@ extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);
 
 extern int has_sha1_pack(const unsigned char *sha1);
 
+extern int has_pack_index(const unsigned char *sha1);
+
 #endif
packfile.c +8 -0
index 507f65236..28a16206c 100644
@@ -1855,3 +1855,11 @@ int has_sha1_pack(const unsigned char *sha1)
 	struct pack_entry e;
 	return find_pack_entry(sha1, &e);
 }
+
+int has_pack_index(const unsigned char *sha1)
+{
+	struct stat st;
+	if (stat(sha1_pack_index_name(sha1), &st))
+		return 0;
+	return 1;
+}
sha1_file.c +0 -8
index 2610ea057..8584f6cf2 100644
@@ -1621,14 +1621,6 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)
 	return ret;
 }
 
-int has_pack_index(const unsigned char *sha1)
-{
-	struct stat st;
-	if (stat(sha1_pack_index_name(sha1), &st))
-		return 0;
-	return 1;
-}
-
 int has_sha1_file_with_flags(const unsigned char *sha1, int flags)
 {
 	if (!startup_info->have_repository)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#48 lore raw

PATCH v2 25/25 pack: move for_each_packed_object()

In reply to Jonathan Tan
---
 builtin/cat-file.c |  1 +
 cache.h            |  7 +------
 pack.h             | 11 +++++++++++
 packfile.c         | 40 ++++++++++++++++++++++++++++++++++++++++
 reachable.c        |  1 +
 sha1_file.c        | 40 ----------------------------------------
 6 files changed, 54 insertions(+), 46 deletions(-)
builtin/cat-file.c +1 -0
index 96b786e48..316ef5c98 100644
@@ -12,6 +12,7 @@
 #include "streaming.h"
 #include "tree-walk.h"
 #include "sha1-array.h"
+#include "pack.h"
 
 struct batch_options {
 	int enabled;
cache.h +1 -6
index 656b39d51..6c3822783 100644
@@ -1660,17 +1660,12 @@ int for_each_loose_file_in_objdir_buf(struct strbuf *path,
 				      void *data);
 
 /*
- * Iterate over loose and packed objects in both the local
+ * Iterate over loose objects in both the local
  * repository and any alternates repositories (unless the
  * LOCAL_ONLY flag is set).
  */
 #define FOR_EACH_OBJECT_LOCAL_ONLY 0x1
-typedef int each_packed_object_fn(const struct object_id *oid,
-				  struct packed_git *pack,
-				  uint32_t pos,
-				  void *data);
 extern int for_each_loose_object(each_loose_object_fn, void *, unsigned flags);
-extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);
 
 struct object_info {
 	/* Request */
pack.h +11 -0
index 2c2a347ba..905b05be5 100644
@@ -227,4 +227,15 @@ extern int has_sha1_pack(const unsigned char *sha1);
 
 extern int has_pack_index(const unsigned char *sha1);
 
+/*
+ * Iterate over packed objects in both the local
+ * repository and any alternates repositories (unless the
+ * FOR_EACH_OBJECT_LOCAL_ONLY flag, defined in cache.h, is set).
+ */
+typedef int each_packed_object_fn(const struct object_id *oid,
+				  struct packed_git *pack,
+				  uint32_t pos,
+				  void *data);
+extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);
+
 #endif
packfile.c +40 -0
index 28a16206c..031a40828 100644
@@ -1863,3 +1863,43 @@ int has_pack_index(const unsigned char *sha1)
 		return 0;
 	return 1;
 }
+
+static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)
+{
+	uint32_t i;
+	int r = 0;
+
+	for (i = 0; i < p->num_objects; i++) {
+		struct object_id oid;
+
+		if (!nth_packed_object_oid(&oid, p, i))
+			return error("unable to get sha1 of object %u in %s",
+				     i, p->pack_name);
+
+		r = cb(&oid, p, i, data);
+		if (r)
+			break;
+	}
+	return r;
+}
+
+int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
+{
+	struct packed_git *p;
+	int r = 0;
+	int pack_errors = 0;
+
+	prepare_packed_git();
+	for (p = packed_git; p; p = p->next) {
+		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
+			continue;
+		if (open_pack_index(p)) {
+			pack_errors = 1;
+			continue;
+		}
+		r = for_each_object_in_pack(p, cb, data);
+		if (r)
+			break;
+	}
+	return r ? r : pack_errors;
+}
reachable.c +1 -0
index c62efbfd4..ef606ae17 100644
@@ -9,6 +9,7 @@
 #include "cache-tree.h"
 #include "progress.h"
 #include "list-objects.h"
+#include "pack.h"
 
 struct connectivity_progress {
 	struct progress *progress;
sha1_file.c +0 -40
index 8584f6cf2..3f3f9174f 100644
@@ -2013,46 +2013,6 @@ int for_each_loose_object(each_loose_object_fn cb, void *data, unsigned flags)
 	return foreach_alt_odb(loose_from_alt_odb, &alt);
 }
 
-static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)
-{
-	uint32_t i;
-	int r = 0;
-
-	for (i = 0; i < p->num_objects; i++) {
-		struct object_id oid;
-
-		if (!nth_packed_object_oid(&oid, p, i))
-			return error("unable to get sha1 of object %u in %s",
-				     i, p->pack_name);
-
-		r = cb(&oid, p, i, data);
-		if (r)
-			break;
-	}
-	return r;
-}
-
-int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
-{
-	struct packed_git *p;
-	int r = 0;
-	int pack_errors = 0;
-
-	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
-		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
-			continue;
-		if (open_pack_index(p)) {
-			pack_errors = 1;
-			continue;
-		}
-		r = for_each_object_in_pack(p, cb, data);
-		if (r)
-			break;
-	}
-	return r ? r : pack_errors;
-}
-
 static int check_stream_sha1(git_zstream *stream,
 			     const char *hdr,
 			     unsigned long size,
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#49 lore raw

Re: [PATCH v2 00/25] Move exported packfile funcs to its own file

In reply to Jonathan Tan
Quoted 16 lines · Jonathan Tan

Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like freshen_packed_object() that are used only by non-pack-specific functions are not moved.

In the end, 3 functions needed to be made global. They are find_pack_entry(), mark_bad_packed_object(), and has_packed_and_bad().

Of the 3, find_pack_entry() is probably legitimately promoted. But I think that the latter two functions needing to be accessed from sha1_file.c points to a design that could be improved - they are only used when packed_object_info() detects corruption, and used for marking as bad and printing messages to the user respectively, which packed_object_info() should probably do itself. But I have not made this change in this patch set.

Quoted 3 lines

(Other than the 3 functions above, there are some variables and functions that are temporarily made global, but reduced back to static when the wide scope is no longer needed.)

I read through the patches yesterday and had no comment.

Thanks,
Stefan

#50 lore raw

Re: [PATCH v2 00/25] Move exported packfile funcs to its own file

In reply to Jonathan Tan
Quoted 4 lines · Jonathan Tan

Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like freshen_packed_object() that are used only by non-pack-specific functions are not moved.

This will interfere with smaller changes and fixes we want to have early in the 'master' branch, so while I think it is a good idea to do something like this in the longer term, I'd have to ask you to either hold on or rebase this on them (you'll know what else you are conflicting with when you try to merge this to 'pu' yourself).

Thanks.

#51 lore raw

Re: [PATCH v2 00/25] Move exported packfile funcs to its own file

Quoted 7 lines · On Thu, 10 Aug 2017 14:19:59 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 4 lines · Jonathan Tan

Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like freshen_packed_object() that are used only by non-pack-specific functions are not moved.

Quoted 7 lines

This will interfere with smaller changes and fixes we want to have early in the 'master' branch, so while I think it is a good idea to do something like this in the longer term, I'd have to ask you to either hold on or rebase this on them (you'll know what else you are conflicting with when you try to merge this to 'pu' yourself).

Thanks.

OK, I'll wait until you have updated the master branch, then I'll try to rebase on it.

#52 lore raw

Re: [PATCH v2 00/25] Move exported packfile funcs to its own file

Quoted 16 lines · Jonathan Tan
Quoted 13 lines · On Thu, 10 Aug 2017 14:19:59 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 4 lines · Jonathan Tan

Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like freshen_packed_object() that are used only by non-pack-specific functions are not moved.

This will interfere with smaller changes and fixes we want to have early in the 'master' branch, so while I think it is a good idea to do something like this in the longer term, I'd have to ask you to either hold on or rebase this on them (you'll know what else you are conflicting with when you try to merge this to 'pu' yourself).

Quoted 4 lines
Quoted 1 line

Thanks.

OK, I'll wait until you have updated the master branch, then I'll try to rebase on it.

That will take a few weeks, and I do not think we want you idling during that time ;-).

You'd need to double check, but I think the topics that cause trouble are rs/find-apck-entry-bisection and jk/drop-sha1-entry-pos; you can start from v2.14.1 and merge these topics on top and then build your change on top. That would allow you to start cooking before both of them graduate to 'master', as I expect they are both quick-to-next material. There might be other topics that interfere with what you are doing, but you can easily find out what they are if you do a trial merge to 'next' and 'pu' yourself.

Thanks.

#53 lore raw

cover non-move patches in preparation for packfile.c

Thanks, I'll work on that.

In the meantime, could these 2 patches be merged in (after review, of course) first? This will make the remaining set much easier to review, as you can check most of them with the new --color-moved functionality of diff.

Incidentally, after rebasing on "pu" and resolving the conflicts, I noticed some "zebra stripes" (from --color-moved) where I didn't expect time. I'll try to figure out that issue before I resend the rest of the patches (which will be after the relevant branches are merged to "next").

Jonathan Tan (2):
sha1_file: set whence in storage-specific info fn
sha1_file: remove read_packed_sha1()

 sha1_file.c | 39 +++++++--------------------------------
 1 file changed, 7 insertions(+), 32 deletions(-)
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#54 lore raw

PATCH 1/2 sha1_file: set whence in storage-specific info fn

Move the setting of oi->whence to sha1_loose_object_info() and packed_object_info().

This allows sha1_object_info_extended() to not need to know about the delta base cache. This will be useful during a future refactoring in which packfile-related functions, including the handling of the delta base cache, will be moved to a separate file.

---
 sha1_file.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
sha1_file.c +6 -7
index b60ae15f7..910109fd9 100644
@@ -2444,6 +2444,9 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,
 			hashclr(oi->delta_base_sha1);
 	}
 
+	oi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :
+							  OI_PACKED;
+
 out:
 	unuse_pack(&w_curs);
 	return type;
@@ -2973,6 +2976,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (oi->sizep == &size_scratch)
 		oi->sizep = NULL;
 	strbuf_release(&hdrbuf);
+	oi->whence = OI_LOOSE;
 	return (status < 0) ? status : 0;
 }
 
@@ -3010,10 +3014,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 
 	if (!find_pack_entry(real, &e)) {
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags)) {
-			oi->whence = OI_LOOSE;
+		if (!sha1_loose_object_info(real, oi, flags))
 			return 0;
-		}
 
 		/* Not a loose object; someone else may have just packed it. */
 		if (flags & OBJECT_INFO_QUICK) {
@@ -3036,10 +3038,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 	if (rtype < 0) {
 		mark_bad_packed_object(e.p, real);
 		return sha1_object_info_extended(real, oi, 0);
-	} else if (in_delta_base_cache(e.p, e.offset)) {
-		oi->whence = OI_DBCACHED;
-	} else {
-		oi->whence = OI_PACKED;
+	} else if (oi->whence == OI_PACKED) {
 		oi->u.packed.offset = e.offset;
 		oi->u.packed.pack = e.p;
 		oi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#55 lore raw

Re: [PATCH 1/2] sha1_file: set whence in storage-specific info fn

Quoted 10 lines · Jonathan Tan

Move the setting of oi->whence to sha1_loose_object_info() and packed_object_info().

This allows sha1_object_info_extended() to not need to know about the delta base cache. This will be useful during a future refactoring in which packfile-related functions, including the handling of the delta base cache, will be moved to a separate file.

---

Output from

    git grep -E -e '(\.|->)whence'

makes me wonder if the oi->whence thing is a bit over-engineered, though. The only real user of this information is the streaming code, which wants to see if it can grab an undeltified deflated data directly out of the pack file (and if so from which packfile at what offset), or if it can open a loose object file and slurp deflated data out of it.

But that is totally outside the scope of this patch. This looks like a safe no-op conversion to me.

Thanks.

Quoted 49 lines
 sha1_file.c | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
sha1_file.c +6 -7
index b60ae15f7..910109fd9 100644
@@ -2444,6 +2444,9 @@ int packed_object_info(struct packed_git *p, off_t obj_offset,
 			hashclr(oi->delta_base_sha1);
 	}
 
+	oi->whence = in_delta_base_cache(p, obj_offset) ? OI_DBCACHED :
+							  OI_PACKED;
+
 out:
 	unuse_pack(&w_curs);
 	return type;
@@ -2973,6 +2976,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (oi->sizep == &size_scratch)
 		oi->sizep = NULL;
 	strbuf_release(&hdrbuf);
+	oi->whence = OI_LOOSE;
 	return (status < 0) ? status : 0;
 }
 
@@ -3010,10 +3014,8 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 
 	if (!find_pack_entry(real, &e)) {
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags)) {
-			oi->whence = OI_LOOSE;
+		if (!sha1_loose_object_info(real, oi, flags))
 			return 0;
-		}
 
 		/* Not a loose object; someone else may have just packed it. */
 		if (flags & OBJECT_INFO_QUICK) {
@@ -3036,10 +3038,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 	if (rtype < 0) {
 		mark_bad_packed_object(e.p, real);
 		return sha1_object_info_extended(real, oi, 0);
-	} else if (in_delta_base_cache(e.p, e.offset)) {
-		oi->whence = OI_DBCACHED;
-	} else {
-		oi->whence = OI_PACKED;
+	} else if (oi->whence == OI_PACKED) {
 		oi->u.packed.offset = e.offset;
 		oi->u.packed.pack = e.p;
 		oi->u.packed.is_delta = (rtype == OBJ_REF_DELTA ||
#56 lore raw

PATCH 2/2 sha1_file: remove read_packed_sha1()

In reply to Jonathan Tan

Use read_object() in its place instead. This avoids duplication of code.

This makes force_object_loose() slightly slower (because of a redundant check of loose object storage), but only in the error case.

---
 sha1_file.c | 26 +-------------------------
 1 file changed, 1 insertion(+), 25 deletions(-)
sha1_file.c +1 -25
index 910109fd9..0f758eabf 100644
@@ -3062,30 +3062,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)
 	return type;
 }
 
-static void *read_packed_sha1(const unsigned char *sha1,
-			      enum object_type *type, unsigned long *size)
-{
-	struct pack_entry e;
-	void *data;
-
-	if (!find_pack_entry(sha1, &e))
-		return NULL;
-	data = cache_or_unpack_entry(e.p, e.offset, size, type);
-	if (!data) {
-		/*
-		 * We're probably in deep shit, but let's try to fetch
-		 * the required object anyway from another pack or loose.
-		 * This should happen only in the presence of a corrupted
-		 * pack, and is better than failing outright.
-		 */
-		error("failed to read object %s at offset %"PRIuMAX" from %s",
-		      sha1_to_hex(sha1), (uintmax_t)e.offset, e.p->pack_name);
-		mark_bad_packed_object(e.p, sha1);
-		data = read_object(sha1, type, size);
-	}
-	return data;
-}
-
 int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,
 		      unsigned char *sha1)
 {
@@ -3468,7 +3444,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)
 
 	if (has_loose_object(sha1))
 		return 0;
-	buf = read_packed_sha1(sha1, &type, &len);
+	buf = read_object(sha1, &type, &len);
 	if (!buf)
 		return error("cannot read sha1_file for %s", sha1_to_hex(sha1));
 	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %lu", typename(type), len) + 1;
Signature · 1 line
2.14.0.434.g98096fd7a8-goog
#57 lore raw

Re: [PATCH 2/2] sha1_file: remove read_packed_sha1()

Quoted 9 lines · Jonathan Tan

Use read_object() in its place instead. This avoids duplication of code.

This makes force_object_loose() slightly slower (because of a redundant check of loose object storage), but only in the error case.

---
 sha1_file.c | 26 +-------------------------
 1 file changed, 1 insertion(+), 25 deletions(-)

The original code insisted on reading from pack and never from a loose object, because it knew it would return early when it found a loose version. Now we allow a loose one to appear in the middle of force_object_loose() operation and happily read from it when we do not see a pack entry for the object---presumably because we are racing with another simultanous repack process, or something?---and then write it out as a new (and identical) loose object, which would not do any harm.

So this is not strictly a no-op conversion; I have a gut feeling that it would make it more robust, not less, in the presence of another racing repack process, but I haven't really thought through race scenarios that may make difference in its behaviour.

Quoted 44 lines
sha1_file.c +1 -25
index 910109fd9..0f758eabf 100644
@@ -3062,30 +3062,6 @@ int sha1_object_info(const unsigned char *sha1, unsigned long *sizep)
 	return type;
 }
 
-static void *read_packed_sha1(const unsigned char *sha1,
-			      enum object_type *type, unsigned long *size)
-{
-	struct pack_entry e;
-	void *data;
-
-	if (!find_pack_entry(sha1, &e))
-		return NULL;
-	data = cache_or_unpack_entry(e.p, e.offset, size, type);
-	if (!data) {
-		/*
-		 * We're probably in deep shit, but let's try to fetch
-		 * the required object anyway from another pack or loose.
-		 * This should happen only in the presence of a corrupted
-		 * pack, and is better than failing outright.
-		 */
-		error("failed to read object %s at offset %"PRIuMAX" from %s",
-		      sha1_to_hex(sha1), (uintmax_t)e.offset, e.p->pack_name);
-		mark_bad_packed_object(e.p, sha1);
-		data = read_object(sha1, type, size);
-	}
-	return data;
-}
-
 int pretend_sha1_file(void *buf, unsigned long len, enum object_type type,
 		      unsigned char *sha1)
 {
@@ -3468,7 +3444,7 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)
 
 	if (has_loose_object(sha1))
 		return 0;
-	buf = read_packed_sha1(sha1, &type, &len);
+	buf = read_object(sha1, &type, &len);
 	if (!buf)
 		return error("cannot read sha1_file for %s", sha1_to_hex(sha1));
 	hdrlen = xsnprintf(hdr, sizeof(hdr), "%s %lu", typename(type), len) + 1;
#58 lore raw

Re: [PATCH v2 00/25] Move exported packfile funcs to its own file

In reply to Jonathan Tan
Quoted 16 lines · Jonathan Tan

Here is the complete patch set. I have only moved the exported functions that operate with packfiles and their static helpers - for example, static functions like freshen_packed_object() that are used only by non-pack-specific functions are not moved.

In the end, 3 functions needed to be made global. They are find_pack_entry(), mark_bad_packed_object(), and has_packed_and_bad().

Of the 3, find_pack_entry() is probably legitimately promoted. But I think that the latter two functions needing to be accessed from sha1_file.c points to a design that could be improved - they are only used when packed_object_info() detects corruption, and used for marking as bad and printing messages to the user respectively, which packed_object_info() should probably do itself. But I have not made this change in this patch set.

Quoted 4 lines

(Other than the 3 functions above, there are some variables and functions that are temporarily made global, but reduced back to static when the wide scope is no longer needed.)

Nice to see the pack file functions being refactored out. I looked at the end result and it looked good to me.

Do you have the energy to do a similar refactoring for the remaining public functions residing in sha1_file.c? Perhaps a new sha1_file.h? It would be nice to get more things out of cache.h. :)

#59 lore raw

Re: [PATCH v2 00/25] Move exported packfile funcs to its own file

Quoted 2 lines · On Fri, 11 Aug 2017 15:41:28 -0400 Ben Peart <[email protected]> wrote:

Nice to see the pack file functions being refactored out. I looked at the end result and it looked good to me.

Thanks.

Quoted 3 lines

Do you have the energy to do a similar refactoring for the remaining public functions residing in sha1_file.c? Perhaps a new sha1_file.h? It would be nice to get more things out of cache.h. :)

I agree that that would be desirable, but for now I'll leave that for someone else to do :-)

#60 lore raw

v3 cover Move exported packfile funcs to its own file

In reply to Jonathan Tan
Quoted 8 lines

You'd need to double check, but I think the topics that cause trouble are rs/find-apck-entry-bisection and jk/drop-sha1-entry-pos; you can start from v2.14.1 and merge these topics on top and then build your change on top. That would allow you to start cooking before both of them graduate to 'master', as I expect they are both quick-to-next material. There might be other topics that interfere with what you are doing, but you can easily find out what they are if you do a trial merge to 'next' and 'pu' yourself.

OK - in addition to the 2 you mentioned, I have found some others (likely added after you wrote that). The complete list is:
- rs/find-pack-entry-bisection
- jk/drop-sha1-entry-pos
- jt/sha1-file-cleanup (formerly part of this set)
- mk/use-size-t-in-zlib
- rs/unpack-entry-leakfix

I have merged all of these and rebased my patches on top.

Other changes:
- Used packfile.h instead of pack.h (following most people's preference)
- Ensured that I added functions to packfile.h retaining the order they were originally in, so that if you run "git diff <base> --color-moved --patience", there are much fewer zebra stripes

The merge base commit can be accessed online [1], if you need it.

[1] https://github.com/jonathantanmy/git/commits/packmigrate

Jonathan Tan (23):
pack: move pack name-related functions
pack: move static state variables
pack: move pack_report()
pack: move open_pack_index(), parse_pack_index()
pack: move release_pack_memory()
pack: move pack-closing functions
pack: move use_pack()
pack: move unuse_pack()
pack: move add_packed_git()
pack: move install_packed_git()
pack: move {,re}prepare_packed_git and approximate_object_count
pack: move unpack_object_header_buffer()
pack: move get_size_from_delta()
pack: move unpack_object_header()
pack: move clear_delta_base_cache(), packed_object_info(),
unpack_entry()
pack: move nth_packed_object_{sha1,oid}
pack: move check_pack_index_ptr(), nth_packed_object_offset()
pack: move find_pack_entry_one(), is_pack_valid()
pack: move find_sha1_pack()
pack: move find_pack_entry() and make it global
pack: move has_sha1_pack()
pack: move has_pack_index()
pack: move for_each_packed_object()

 Makefile                 |    1 +
 builtin/am.c             |    1 +
 builtin/cat-file.c       |    1 +
 builtin/clone.c          |    1 +
 builtin/count-objects.c  |    1 +
 builtin/fetch.c          |    1 +
 builtin/fsck.c           |    1 +
 builtin/gc.c             |    1 +
 builtin/index-pack.c     |    1 +
 builtin/merge.c          |    1 +
 builtin/pack-objects.c   |    1 +
 builtin/pack-redundant.c |    1 +
 builtin/prune-packed.c   |    1 +
 builtin/receive-pack.c   |    1 +
 bulk-checkin.c           |    1 +
 cache.h                  |  122 +--
 connected.c              |    1 +
 diff.c                   |    1 +
 fast-import.c            |    1 +
 fetch-pack.c             |    1 +
 git-compat-util.h        |    2 -
 http-backend.c           |    1 +
 http-push.c              |    1 +
 http-walker.c            |    1 +
 http.c                   |    1 +
 outgoing/packfile.h      |    0
 pack-bitmap.c            |    1 +
 pack-check.c             |    1 +
 packfile.c               | 1896 +++++++++++++++++++++++++++++++++++
 packfile.h               |  138 +++
 path.c                   |    1 +
 reachable.c              |    1 +
 revision.c               |    1 +
 server-info.c            |    1 +
 sha1_file.c              | 2452 ++++++----------------------------------------
 sha1_name.c              |    1 +
 streaming.c              |    1 +
 37 files changed, 2354 insertions(+), 2287 deletions(-)
 create mode 100644 outgoing/packfile.h
 create mode 100644 packfile.c
 create mode 100644 packfile.h
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#61 lore raw

PATCH v3 1/23 pack: move pack name-related functions

Currently, sha1_file.c and cache.h contain many functions, both related to and unrelated to packfiles. This makes both files very large and causes an unclear separation of concerns.

Create a new file, packfile.c, to hold all packfile-related functions currently in sha1_file.c. It has a corresponding header packfile.h.

In this commit, the pack name-related functions are moved. Subsequent commits will move the other functions.

---
 Makefile                 |  1 +
 builtin/index-pack.c     |  1 +
 builtin/pack-redundant.c |  1 +
 cache.h                  | 23 -----------------------
 fast-import.c            |  1 +
 http.c                   |  1 +
 outgoing/packfile.h      |  0
 packfile.c               | 23 +++++++++++++++++++++++
 packfile.h               | 27 +++++++++++++++++++++++++++
 sha1_file.c              | 23 +----------------------
 10 files changed, 56 insertions(+), 45 deletions(-)
 create mode 100644 outgoing/packfile.h
 create mode 100644 packfile.c
 create mode 100644 packfile.h
Makefile +1 -0
index 461c845d3..5cdecaa17 100644
@@ -816,6 +816,7 @@ LIB_OBJS += notes-merge.o
 LIB_OBJS += notes-utils.o
 LIB_OBJS += object.o
 LIB_OBJS += oidset.o
+LIB_OBJS += packfile.o
 LIB_OBJS += pack-bitmap.o
 LIB_OBJS += pack-bitmap-write.o
 LIB_OBJS += pack-check.o
builtin/index-pack.c +1 -0
index 26828c1d8..f2be145e1 100644
@@ -12,6 +12,7 @@
 #include "exec_cmd.h"
 #include "streaming.h"
 #include "thread-utils.h"
+#include "packfile.h"
 
 static const char index_pack_usage[] =
 "git index-pack [-v] [-o <index-file>] [--keep | --keep=<msg>] [--verify] [--strict] (<pack-file> | --stdin [--fix-thin] [<pack-file>])";
builtin/pack-redundant.c +1 -0
index cb1df1c76..aaa813632 100644
@@ -7,6 +7,7 @@
 */
 
 #include "builtin.h"
+#include "packfile.h"
 
 #define BLKSIZE 512
 
cache.h +0 -23
index fcba87a69..aa2b4d390 100644
@@ -902,20 +902,6 @@ extern void check_repository_format(void);
  */
 extern const char *sha1_file_name(const unsigned char *sha1);
 
-/*
- * Return the name of the (local) packfile with the specified sha1 in
- * its name.  The return value is a pointer to memory that is
- * overwritten each time this function is called.
- */
-extern char *sha1_pack_name(const unsigned char *sha1);
-
-/*
- * Return the name of the (local) pack index file with the specified
- * sha1 in its name.  The return value is a pointer to memory that is
- * overwritten each time this function is called.
- */
-extern char *sha1_pack_index_name(const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
@@ -1656,15 +1642,6 @@ extern void pack_report(void);
  */
 extern int odb_mkstemp(struct strbuf *template, const char *pattern);
 
-/*
- * Generate the filename to be used for a pack file with checksum "sha1" and
- * extension "ext". The result is written into the strbuf "buf", overwriting
- * any existing contents. A pointer to buf->buf is returned as a convenience.
- *
- * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx"
- */
-extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);
-
 /*
  * Create a pack .keep file named "name" (which should generally be the output
  * of odb_pack_name). Returns a file descriptor opened for writing, or -1 on
fast-import.c +1 -0
index a959161b4..49516d60e 100644
@@ -167,6 +167,7 @@ Format of STDIN stream:
 #include "quote.h"
 #include "dir.h"
 #include "run-command.h"
+#include "packfile.h"
 
 #define PACK_ID_BITS 16
 #define MAX_PACK_ID ((1<<PACK_ID_BITS)-1)
http.c +1 -0
index c6c010f88..1d1e4ab01 100644
@@ -11,6 +11,7 @@
 #include "pkt-line.h"
 #include "gettext.h"
 #include "transport.h"
+#include "packfile.h"
 
 static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);
 #if LIBCURL_VERSION_NUM >= 0x070a08
outgoing/packfile.h new file +0 -0
new file mode 100644
index 000000000..e69de29bb
packfile.c new file +23 -0
new file mode 100644
index 000000000..0d191dfd6
@@ -0,0 +1,23 @@
+#include "cache.h"
+
+char *odb_pack_name(struct strbuf *buf,
+		    const unsigned char *sha1,
+		    const char *ext)
+{
+	strbuf_reset(buf);
+	strbuf_addf(buf, "%s/pack/pack-%s.%s", get_object_directory(),
+		    sha1_to_hex(sha1), ext);
+	return buf->buf;
+}
+
+char *sha1_pack_name(const unsigned char *sha1)
+{
+	static struct strbuf buf = STRBUF_INIT;
+	return odb_pack_name(&buf, sha1, "pack");
+}
+
+char *sha1_pack_index_name(const unsigned char *sha1)
+{
+	static struct strbuf buf = STRBUF_INIT;
+	return odb_pack_name(&buf, sha1, "idx");
+}
packfile.h new file +27 -0
new file mode 100644
index 000000000..3c4a0dbd7
@@ -0,0 +1,27 @@
+#ifndef PACKFILE_H
+#define PACKFILE_H
+
+/*
+ * Generate the filename to be used for a pack file with checksum "sha1" and
+ * extension "ext". The result is written into the strbuf "buf", overwriting
+ * any existing contents. A pointer to buf->buf is returned as a convenience.
+ *
+ * Example: odb_pack_name(out, sha1, "idx") => ".git/objects/pack/pack-1234..idx"
+ */
+extern char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, const char *ext);
+
+/*
+ * Return the name of the (local) packfile with the specified sha1 in
+ * its name.  The return value is a pointer to memory that is
+ * overwritten each time this function is called.
+ */
+extern char *sha1_pack_name(const unsigned char *sha1);
+
+/*
+ * Return the name of the (local) pack index file with the specified
+ * sha1 in its name.  The return value is a pointer to memory that is
+ * overwritten each time this function is called.
+ */
+extern char *sha1_pack_index_name(const unsigned char *sha1);
+
+#endif
sha1_file.c +1 -22
index c888d7e5b..6e7a20b52 100644
@@ -28,6 +28,7 @@
 #include "list.h"
 #include "mergesort.h"
 #include "quote.h"
+#include "packfile.h"
 
 #define SZ_FMT PRIuMAX
 static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -278,28 +279,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
- char *odb_pack_name(struct strbuf *buf,
-		     const unsigned char *sha1,
-		     const char *ext)
-{
-	strbuf_reset(buf);
-	strbuf_addf(buf, "%s/pack/pack-%s.%s", get_object_directory(),
-		    sha1_to_hex(sha1), ext);
-	return buf->buf;
-}
-
-char *sha1_pack_name(const unsigned char *sha1)
-{
-	static struct strbuf buf = STRBUF_INIT;
-	return odb_pack_name(&buf, sha1, "pack");
-}
-
-char *sha1_pack_index_name(const unsigned char *sha1)
-{
-	static struct strbuf buf = STRBUF_INIT;
-	return odb_pack_name(&buf, sha1, "idx");
-}
-
 struct alternate_object_database *alt_odb_list;
 static struct alternate_object_database **alt_odb_tail;
 
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#62 lore raw

PATCH v3 2/23 pack: move static state variables

In reply to Jonathan Tan

sha1_file.c declares some static variables that store packfile-related state. Move them to packfile.c.

They are temporarily made global, but subsequent commits will restore their scope back to static.

---
 packfile.c  | 14 ++++++++++++++
 packfile.h  |  9 +++++++++
 sha1_file.c | 13 -------------
 3 files changed, 23 insertions(+), 13 deletions(-)
packfile.c +14 -0
index 0d191dfd6..0f46e0617 100644
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "mru.h"
 
 char *odb_pack_name(struct strbuf *buf,
 		    const unsigned char *sha1,
@@ -21,3 +22,16 @@ char *sha1_pack_index_name(const unsigned char *sha1)
 	static struct strbuf buf = STRBUF_INIT;
 	return odb_pack_name(&buf, sha1, "idx");
 }
+
+unsigned int pack_used_ctr;
+unsigned int pack_mmap_calls;
+unsigned int peak_pack_open_windows;
+unsigned int pack_open_windows;
+unsigned int pack_open_fds;
+unsigned int pack_max_fds;
+size_t peak_pack_mapped;
+size_t pack_mapped;
+struct packed_git *packed_git;
+
+static struct mru packed_git_mru_storage;
+struct mru *packed_git_mru = &packed_git_mru_storage;
packfile.h +9 -0
index 3c4a0dbd7..a76bb7cec 100644
@@ -24,4 +24,13 @@ extern char *sha1_pack_name(const unsigned char *sha1);
  */
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 
+extern unsigned int pack_used_ctr;
+extern unsigned int pack_mmap_calls;
+extern unsigned int peak_pack_open_windows;
+extern unsigned int pack_open_windows;
+extern unsigned int pack_open_fds;
+extern unsigned int pack_max_fds;
+extern size_t peak_pack_mapped;
+extern size_t pack_mapped;
+
 #endif
sha1_file.c +0 -13
index 6e7a20b52..2b5ce9959 100644
@@ -683,19 +683,6 @@ static int has_loose_object(const unsigned char *sha1)
 	return check_and_freshen(sha1, 0);
 }
 
-static unsigned int pack_used_ctr;
-static unsigned int pack_mmap_calls;
-static unsigned int peak_pack_open_windows;
-static unsigned int pack_open_windows;
-static unsigned int pack_open_fds;
-static unsigned int pack_max_fds;
-static size_t peak_pack_mapped;
-static size_t pack_mapped;
-struct packed_git *packed_git;
-
-static struct mru packed_git_mru_storage;
-struct mru *packed_git_mru = &packed_git_mru_storage;
-
 void pack_report(void)
 {
 	fprintf(stderr,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#63 lore raw

PATCH v3 3/23 pack: move pack_report()

In reply to Jonathan Tan
---
 cache.h     |  2 --
 packfile.c  | 24 ++++++++++++++++++++++++
 packfile.h  |  2 ++
 sha1_file.c | 24 ------------------------
 4 files changed, 26 insertions(+), 26 deletions(-)
cache.h +0 -2
index aa2b4d390..a0497d469 100644
@@ -1632,8 +1632,6 @@ unsigned long approximate_object_count(void);
 extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
 					 struct packed_git *packs);
 
-extern void pack_report(void);
-
 /*
  * Create a temporary file rooted in the object database directory, or
  * die on failure. The filename is taken from "pattern", which should have the
packfile.c +24 -0
index 0f46e0617..60d9fc3b0 100644
@@ -35,3 +35,27 @@ struct packed_git *packed_git;
 
 static struct mru packed_git_mru_storage;
 struct mru *packed_git_mru = &packed_git_mru_storage;
+
+#define SZ_FMT PRIuMAX
+static inline uintmax_t sz_fmt(size_t s) { return s; }
+
+void pack_report(void)
+{
+	fprintf(stderr,
+		"pack_report: getpagesize()            = %10" SZ_FMT "\n"
+		"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n"
+		"pack_report: core.packedGitLimit      = %10" SZ_FMT "\n",
+		sz_fmt(getpagesize()),
+		sz_fmt(packed_git_window_size),
+		sz_fmt(packed_git_limit));
+	fprintf(stderr,
+		"pack_report: pack_used_ctr            = %10u\n"
+		"pack_report: pack_mmap_calls          = %10u\n"
+		"pack_report: pack_open_windows        = %10u / %10u\n"
+		"pack_report: pack_mapped              = "
+			"%10" SZ_FMT " / %10" SZ_FMT "\n",
+		pack_used_ctr,
+		pack_mmap_calls,
+		pack_open_windows, peak_pack_open_windows,
+		sz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));
+}
packfile.h +2 -0
index a76bb7cec..bfa94c8fe 100644
@@ -33,4 +33,6 @@ extern unsigned int pack_max_fds;
 extern size_t peak_pack_mapped;
 extern size_t pack_mapped;
 
+extern void pack_report(void);
+
 #endif
sha1_file.c +0 -24
index 2b5ce9959..f7c8152ac 100644
@@ -30,9 +30,6 @@
 #include "quote.h"
 #include "packfile.h"
 
-#define SZ_FMT PRIuMAX
-static inline uintmax_t sz_fmt(size_t s) { return s; }
-
 const unsigned char null_sha1[20];
 const struct object_id null_oid;
 const struct object_id empty_tree_oid = {
@@ -683,27 +680,6 @@ static int has_loose_object(const unsigned char *sha1)
 	return check_and_freshen(sha1, 0);
 }
 
-void pack_report(void)
-{
-	fprintf(stderr,
-		"pack_report: getpagesize()            = %10" SZ_FMT "\n"
-		"pack_report: core.packedGitWindowSize = %10" SZ_FMT "\n"
-		"pack_report: core.packedGitLimit      = %10" SZ_FMT "\n",
-		sz_fmt(getpagesize()),
-		sz_fmt(packed_git_window_size),
-		sz_fmt(packed_git_limit));
-	fprintf(stderr,
-		"pack_report: pack_used_ctr            = %10u\n"
-		"pack_report: pack_mmap_calls          = %10u\n"
-		"pack_report: pack_open_windows        = %10u / %10u\n"
-		"pack_report: pack_mapped              = "
-			"%10" SZ_FMT " / %10" SZ_FMT "\n",
-		pack_used_ctr,
-		pack_mmap_calls,
-		pack_open_windows, peak_pack_open_windows,
-		sz_fmt(pack_mapped), sz_fmt(peak_pack_mapped));
-}
-
 /*
  * Open and mmap the index file at path, perform a couple of
  * consistency checks, then record its information to p.  Return 0 on
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#64 lore raw

PATCH v3 4/23 pack: move open_pack_index(), parse_pack_index()

In reply to Jonathan Tan

alloc_packed_git() in packfile.c is duplicated from sha1_file.c. In a subsequent commit, alloc_packed_git() will be removed from sha1_file.c.

---
 builtin/count-objects.c |   1 +
 builtin/fsck.c          |   1 +
 builtin/pack-objects.c  |   1 +
 cache.h                 |   8 ---
 pack-bitmap.c           |   1 +
 pack-check.c            |   1 +
 packfile.c              | 149 ++++++++++++++++++++++++++++++++++++++++++++++++
 packfile.h              |   8 +++
 sha1_file.c             | 140 ---------------------------------------------
 sha1_name.c             |   1 +
 10 files changed, 163 insertions(+), 148 deletions(-)
builtin/count-objects.c +1 -0
index 1d82e61f2..33343818c 100644
@@ -10,6 +10,7 @@
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
+#include "packfile.h"
 
 static unsigned long garbage;
 static off_t size_garbage;
builtin/fsck.c +1 -0
index 99dea7adf..c56207b21 100644
@@ -15,6 +15,7 @@
 #include "progress.h"
 #include "streaming.h"
 #include "decorate.h"
+#include "packfile.h"
 
 #define REACHABLE 0x0001
 #define SEEN      0x0002
builtin/pack-objects.c +1 -0
index 5c5d3d507..08f05cb84 100644
@@ -25,6 +25,7 @@
 #include "sha1-array.h"
 #include "argv-array.h"
 #include "mru.h"
+#include "packfile.h"
 
 static const char *pack_usage[] = {
 	N_("git pack-objects --stdout [<options>...] [< <ref-list> | < <object-list>]"),
cache.h +0 -8
index a0497d469..f271033db 100644
@@ -1611,8 +1611,6 @@ struct pack_entry {
 	struct packed_git *p;
 };
 
-extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
-
 /* A hook to report invalid files in pack directory */
 #define PACKDIR_FILE_PACK 1
 #define PACKDIR_FILE_IDX 2
@@ -1647,12 +1645,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * mmap the index file for the specified packfile (if it is not
- * already mmapped).  Return 0 on success.
- */
-extern int open_pack_index(struct packed_git *);
-
 /*
  * munmap the index file for the specified packfile (if it is
  * currently mmapped).
pack-bitmap.c +1 -0
index 327634cd7..cb3d14ba4 100644
@@ -9,6 +9,7 @@
 #include "pack-bitmap.h"
 #include "pack-revindex.h"
 #include "pack-objects.h"
+#include "packfile.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
pack-check.c +1 -0
index 84469168a..2086f5bb7 100644
@@ -2,6 +2,7 @@
 #include "pack.h"
 #include "pack-revindex.h"
 #include "progress.h"
+#include "packfile.h"
 
 struct idx_entry {
 	off_t                offset;
packfile.c +149 -0

157 lines not on this page · raw message

packfile.h +8 -0
index bfa94c8fe..703887d41 100644
@@ -33,6 +33,14 @@ extern unsigned int pack_max_fds;
 extern size_t peak_pack_mapped;
 extern size_t pack_mapped;
 
+extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
+
 extern void pack_report(void);
 
+/*
+ * mmap the index file for the specified packfile (if it is not
+ * already mmapped).  Return 0 on success.
+ */
+extern int open_pack_index(struct packed_git *);
+
 #endif
sha1_file.c +0 -140

152 lines not on this page · raw message

sha1_name.c +1 -0
index 74fcb6d78..ae237809c 100644
@@ -9,6 +9,7 @@
 #include "remote.h"
 #include "dir.h"
 #include "sha1-array.h"
+#include "packfile.h"
 
 static int get_sha1_oneline(const char *, unsigned char *, struct commit_list *);
 
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#65 lore raw

PATCH v3 5/23 pack: move release_pack_memory()

In reply to Jonathan Tan

The function unuse_one_window() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.

---
 git-compat-util.h |  2 --
 packfile.c        | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 packfile.h        |  4 ++++
 sha1_file.c       | 49 -------------------------------------------------
 4 files changed, 53 insertions(+), 51 deletions(-)
git-compat-util.h +0 -2
index db9c22de7..201056e2d 100644
@@ -749,8 +749,6 @@ const char *inet_ntop(int af, const void *src, char *dst, size_t size);
 extern int git_atexit(void (*handler)(void));
 #endif
 
-extern void release_pack_memory(size_t);
-
 typedef void (*try_to_free_t)(size_t);
 extern try_to_free_t set_try_to_free_routine(try_to_free_t);
 
packfile.c +49 -0

52 lines not on this page · raw message

packfile.h +4 -0
index 703887d41..f6fe1c741 100644
@@ -43,4 +43,8 @@ extern void pack_report(void);
  */
 extern int open_pack_index(struct packed_git *);
 
+extern int unuse_one_window(struct packed_git *current);
+
+extern void release_pack_memory(size_t);
+
 #endif
sha1_file.c +0 -49

55 lines not on this page · raw message

Signature · 1 line
2.14.1.480.gb18f417b89-goog
#66 lore raw

PATCH v3 6/23 pack: move pack-closing functions

In reply to Jonathan Tan

The function close_pack_fd() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.

---
 builtin/am.c           |  1 +
 builtin/clone.c        |  1 +
 builtin/fetch.c        |  1 +
 builtin/merge.c        |  1 +
 builtin/receive-pack.c |  1 +
 cache.h                |  8 --------
 packfile.c             | 54 +++++++++++++++++++++++++++++++++++++++++++++++++
 packfile.h             | 11 ++++++++++
 sha1_file.c            | 55 --------------------------------------------------
 9 files changed, 70 insertions(+), 63 deletions(-)
builtin/am.c +1 -0
index c973bd96d..b9d9ff203 100644
@@ -31,6 +31,7 @@
 #include "mailinfo.h"
 #include "apply.h"
 #include "string-list.h"
+#include "packfile.h"
 
 /**
  * Returns 1 if the file is empty or does not exist, 0 otherwise.
builtin/clone.c +1 -0
index 08b5cc433..13abb075a 100644
@@ -25,6 +25,7 @@
 #include "remote.h"
 #include "run-command.h"
 #include "connected.h"
+#include "packfile.h"
 
 /*
  * Overall FIXMEs:
builtin/fetch.c +1 -0
index c87e59f3b..c86c36f37 100644
@@ -17,6 +17,7 @@
 #include "connected.h"
 #include "argv-array.h"
 #include "utf8.h"
+#include "packfile.h"
 
 static const char * const builtin_fetch_usage[] = {
 	N_("git fetch [<options>] [<repository> [<refspec>...]]"),
builtin/merge.c +1 -0
index 900bafdb4..45e673dcc 100644
@@ -32,6 +32,7 @@
 #include "gpg-interface.h"
 #include "sequencer.h"
 #include "string-list.h"
+#include "packfile.h"
 
 #define DEFAULT_TWOHEAD (1<<0)
 #define DEFAULT_OCTOPUS (1<<1)
builtin/receive-pack.c +1 -0
index cabdc55e0..0019a484f 100644
@@ -23,6 +23,7 @@
 #include "fsck.h"
 #include "tmp-objdir.h"
 #include "oidset.h"
+#include "packfile.h"
 
 static const char * const receive_pack_usage[] = {
 	N_("git receive-pack <git-dir>"),
cache.h +0 -8
index f271033db..bbc56566e 100644
@@ -1645,15 +1645,7 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * munmap the index file for the specified packfile (if it is
- * currently mmapped).
- */
-extern void close_pack_index(struct packed_git *);
-
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, size_t *);
-extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
packfile.c +54 -0

57 lines not on this page · raw message

packfile.h +11 -0
index f6fe1c741..c6a07de62 100644
@@ -43,6 +43,17 @@ extern void pack_report(void);
  */
 extern int open_pack_index(struct packed_git *);
 
+/*
+ * munmap the index file for the specified packfile (if it is
+ * currently mmapped).
+ */
+extern void close_pack_index(struct packed_git *);
+
+extern void close_pack_windows(struct packed_git *);
+extern void close_all_packs(void);
+
+extern int close_pack_fd(struct packed_git *);
+
 extern int unuse_one_window(struct packed_git *current);
 
 extern void release_pack_memory(size_t);
sha1_file.c +0 -55

67 lines not on this page · raw message

Signature · 1 line
2.14.1.480.gb18f417b89-goog
#67 lore raw

PATCH v3 7/23 pack: move use_pack()

In reply to Jonathan Tan

The function open_packed_git() needs to be temporarily made global. Its scope will be restored to static in a subsequent commit.

---
 cache.h     |   1 -
 packfile.c  | 303 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--
 packfile.h  |  14 +--
 sha1_file.c | 285 --------------------------------------------------------
 streaming.c |   1 +
 5 files changed, 298 insertions(+), 306 deletions(-)
cache.h +0 -1
index bbc56566e..a27018210 100644
@@ -1645,7 +1645,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, size_t *);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
packfile.c +294 -9

325 lines not on this page · raw message

packfile.h +3 -11
index c6a07de62..97cfc5e70 100644
@@ -24,14 +24,7 @@ extern char *sha1_pack_name(const unsigned char *sha1);
  */
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 
-extern unsigned int pack_used_ctr;
-extern unsigned int pack_mmap_calls;
-extern unsigned int peak_pack_open_windows;
-extern unsigned int pack_open_windows;
 extern unsigned int pack_open_fds;
-extern unsigned int pack_max_fds;
-extern size_t peak_pack_mapped;
-extern size_t pack_mapped;
 
 extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
 
@@ -49,13 +42,12 @@ extern int open_pack_index(struct packed_git *);
  */
 extern void close_pack_index(struct packed_git *);
 
+extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, size_t *);
 extern void close_pack_windows(struct packed_git *);
 extern void close_all_packs(void);
 
-extern int close_pack_fd(struct packed_git *);
-
-extern int unuse_one_window(struct packed_git *current);
-
 extern void release_pack_memory(size_t);
 
+extern int open_packed_git(struct packed_git *p);
+
 #endif
sha1_file.c +0 -285

297 lines not on this page · raw message

streaming.c +1 -0
index 9afa66b8b..6f1c60f12 100644
@@ -3,6 +3,7 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "packfile.h"
 
 enum input_source {
 	stream_error = -1,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#68 lore raw

PATCH v3 8/23 pack: move unuse_pack()

In reply to Jonathan Tan
---
 cache.h     | 1 -
 packfile.c  | 9 +++++++++
 packfile.h  | 1 +
 sha1_file.c | 9 ---------
 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -1
index a27018210..0313b0b8d 100644
@@ -1645,7 +1645,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
packfile.c +9 -0
index ea451d27e..0c97c3a1a 100644
@@ -596,3 +596,12 @@ unsigned char *use_pack(struct packed_git *p,
 		*left = win->len - xsize_t(offset);
 	return win->base + offset;
 }
+
+void unuse_pack(struct pack_window **w_cursor)
+{
+	struct pack_window *w = *w_cursor;
+	if (w) {
+		w->inuse_cnt--;
+		*w_cursor = NULL;
+	}
+}
packfile.h +1 -0
index 97cfc5e70..b5db490ab 100644
@@ -45,6 +45,7 @@ extern void close_pack_index(struct packed_git *);
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, size_t *);
 extern void close_pack_windows(struct packed_git *);
 extern void close_all_packs(void);
+extern void unuse_pack(struct pack_window **);
 
 extern void release_pack_memory(size_t);
 
sha1_file.c +0 -9
index 7704801d1..84d96d0ab 100644
@@ -718,15 +718,6 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
-void unuse_pack(struct pack_window **w_cursor)
-{
-	struct pack_window *w = *w_cursor;
-	if (w) {
-		w->inuse_cnt--;
-		*w_cursor = NULL;
-	}
-}
-
 static struct packed_git *alloc_packed_git(int extra)
 {
 	struct packed_git *p = xmalloc(st_add(sizeof(*p), extra));
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#69 lore raw

PATCH v3 9/23 pack: move add_packed_git()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 connected.c |  1 +
 packfile.c  | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 packfile.h  |  1 +
 sha1_file.c | 61 -------------------------------------------------------------
 5 files changed, 55 insertions(+), 62 deletions(-)
cache.h +0 -1
index 0313b0b8d..3625509f9 100644
@@ -1646,7 +1646,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
 extern int odb_pack_keep(const char *name);
 
 extern void clear_delta_base_cache(void);
-extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
 /*
  * Make sure that a pointer access into an mmap'd index file is within bounds,
connected.c +1 -0
index 136c2ac16..3e3f0148c 100644
@@ -3,6 +3,7 @@
 #include "sigchain.h"
 #include "connected.h"
 #include "transport.h"
+#include "pack.h"
 
 /*
  * If we feed all the commits we want to verify to this command
packfile.c +53 -0

56 lines not on this page · raw message

packfile.h +1 -0
index b5db490ab..1e932a49e 100644
@@ -46,6 +46,7 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t
 extern void close_pack_windows(struct packed_git *);
 extern void close_all_packs(void);
 extern void unuse_pack(struct pack_window **);
+extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
 extern void release_pack_memory(size_t);
 
sha1_file.c +0 -61

67 lines not on this page · raw message

Signature · 1 line
2.14.1.480.gb18f417b89-goog
#70 lore raw

PATCH v3 10/23 pack: move install_packed_git()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 packfile.c  | 11 ++++++++++-
 packfile.h  |  2 ++
 sha1_file.c |  9 ---------
 4 files changed, 12 insertions(+), 11 deletions(-)
cache.h +0 -1
index 3625509f9..c4d8bee52 100644
@@ -1619,7 +1619,6 @@ extern void (*report_garbage)(unsigned seen_bits, const char *path);
 
 extern void prepare_packed_git(void);
 extern void reprepare_packed_git(void);
-extern void install_packed_git(struct packed_git *pack);
 
 /*
  * Give a rough count of objects in the repository. This sacrifices accuracy
packfile.c +10 -1
index d1433d8c7..9a65aa4f6 100644
@@ -28,7 +28,7 @@ static unsigned int pack_used_ctr;
 static unsigned int pack_mmap_calls;
 static unsigned int peak_pack_open_windows;
 static unsigned int pack_open_windows;
-unsigned int pack_open_fds;
+static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
@@ -658,3 +658,12 @@ struct packed_git *add_packed_git(const char *path, size_t path_len, int local)
 		hashclr(p->sha1);
 	return p;
 }
+
+void install_packed_git(struct packed_git *pack)
+{
+	if (pack->pack_fd != -1)
+		pack_open_fds++;
+
+	pack->next = packed_git;
+	packed_git = pack;
+}
packfile.h +2 -0
index 1e932a49e..a18029184 100644
@@ -28,6 +28,8 @@ extern unsigned int pack_open_fds;
 
 extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
 
+extern void install_packed_git(struct packed_git *pack);
+
 extern void pack_report(void);
 
 /*
sha1_file.c +0 -9
index 0929fc10e..b77e7e3c3 100644
@@ -718,15 +718,6 @@ void *xmmap(void *start, size_t length,
 	return ret;
 }
 
-void install_packed_git(struct packed_git *pack)
-{
-	if (pack->pack_fd != -1)
-		pack_open_fds++;
-
-	pack->next = packed_git;
-	packed_git = pack;
-}
-
 void (*report_garbage)(unsigned seen_bits, const char *path);
 
 static void report_helper(const struct string_list *list,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#71 lore raw

PATCH v3 11/23 pack: move {,re}prepare_packed_git and approximate_object_count

In reply to Jonathan Tan
---
 builtin/gc.c   |   1 +
 bulk-checkin.c |   1 +
 cache.h        |  15 ----
 connected.c    |   2 +-
 fetch-pack.c   |   1 +
 http-backend.c |   1 +
 packfile.c     | 217 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 packfile.h     |  16 ++++-
 path.c         |   1 +
 server-info.c  |   1 +
 sha1_file.c    | 214 --------------------------------------------------------
 11 files changed, 238 insertions(+), 232 deletions(-)
builtin/gc.c +1 -0
index e6b84475a..3c78fcb9b 100644
@@ -19,6 +19,7 @@
 #include "sigchain.h"
 #include "argv-array.h"
 #include "commit.h"
+#include "packfile.h"
 
 #define FAILED_RUN "failed to run %s"
 
bulk-checkin.c +1 -0
index 5be7ce5c7..9a1f6c49a 100644
@@ -6,6 +6,7 @@
 #include "csum-file.h"
 #include "pack.h"
 #include "strbuf.h"
+#include "packfile.h"
 
 static struct bulk_checkin_state {
 	unsigned plugged:1;
cache.h +0 -15
index c4d8bee52..63765d481 100644
@@ -1611,21 +1611,6 @@ struct pack_entry {
 	struct packed_git *p;
 };
 
-/* A hook to report invalid files in pack directory */
-#define PACKDIR_FILE_PACK 1
-#define PACKDIR_FILE_IDX 2
-#define PACKDIR_FILE_GARBAGE 4
-extern void (*report_garbage)(unsigned seen_bits, const char *path);
-
-extern void prepare_packed_git(void);
-extern void reprepare_packed_git(void);
-
-/*
- * Give a rough count of objects in the repository. This sacrifices accuracy
- * for speed.
- */
-unsigned long approximate_object_count(void);
-
 extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
 					 struct packed_git *packs);
 
connected.c +1 -1
index 3e3f0148c..f416b0505 100644
@@ -3,7 +3,7 @@
 #include "sigchain.h"
 #include "connected.h"
 #include "transport.h"
-#include "pack.h"
+#include "packfile.h"
 
 /*
  * If we feed all the commits we want to verify to this command
fetch-pack.c +1 -0
index fbbc99c88..105506e9a 100644
@@ -17,6 +17,7 @@
 #include "prio-queue.h"
 #include "sha1-array.h"
 #include "oidset.h"
+#include "packfile.h"
 
 static int transfer_unpack_limit = -1;
 static int fetch_unpack_limit = -1;
http-backend.c +1 -0
index 519025d2c..8076b1d5e 100644
@@ -9,6 +9,7 @@
 #include "string-list.h"
 #include "url.h"
 #include "argv-array.h"
+#include "packfile.h"
 
 static const char content_type[] = "Content-Type";
 static const char content_length[] = "Content-Length";
packfile.c +217 -0

226 lines not on this page · raw message

packfile.h +14 -2
index a18029184..1cfda1d00 100644
@@ -24,12 +24,24 @@ extern char *sha1_pack_name(const unsigned char *sha1);
  */
 extern char *sha1_pack_index_name(const unsigned char *sha1);
 
-extern unsigned int pack_open_fds;
-
 extern struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path);
 
+/* A hook to report invalid files in pack directory */
+#define PACKDIR_FILE_PACK 1
+#define PACKDIR_FILE_IDX 2
+#define PACKDIR_FILE_GARBAGE 4
+extern void (*report_garbage)(unsigned seen_bits, const char *path);
+
+extern void prepare_packed_git(void);
+extern void reprepare_packed_git(void);
 extern void install_packed_git(struct packed_git *pack);
 
+/*
+ * Give a rough count of objects in the repository. This sacrifices accuracy
+ * for speed.
+ */
+unsigned long approximate_object_count(void);
+
 extern void pack_report(void);
 
 /*
path.c +1 -0
index e50d2befc..b533ec938 100644
@@ -9,6 +9,7 @@
 #include "worktree.h"
 #include "submodule-config.h"
 #include "path.h"
+#include "packfile.h"
 
 static int get_st_mode_bits(const char *path, int *mode)
 {
server-info.c +1 -0
index 5ec5b1d82..26a6c20b7 100644
@@ -3,6 +3,7 @@
 #include "object.h"
 #include "commit.h"
 #include "tag.h"
+#include "packfile.h"
 
 /*
  * Create the file "path" by writing to a temporary file and renaming
sha1_file.c +0 -214

220 lines not on this page · raw message

Signature · 1 line
2.14.1.480.gb18f417b89-goog
#72 lore raw

PATCH v3 12/23 pack: move unpack_object_header_buffer()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 packfile.c  | 25 +++++++++++++++++++++++++
 packfile.h  |  2 ++
 sha1_file.c | 25 -------------------------
 4 files changed, 27 insertions(+), 26 deletions(-)
cache.h +0 -1
index 63765d481..75cc0c497 100644
@@ -1669,7 +1669,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)
 
 extern int is_pack_valid(struct packed_git *);
 extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
-extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
 extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
 extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
 
packfile.c +25 -0
index 9cf462856..43b708812 100644
@@ -884,3 +884,28 @@ void reprepare_packed_git(void)
 	prepare_packed_git_run_once = 0;
 	prepare_packed_git();
 }
+
+unsigned long unpack_object_header_buffer(const unsigned char *buf,
+		unsigned long len, enum object_type *type, unsigned long *sizep)
+{
+	unsigned shift;
+	unsigned long size, c;
+	unsigned long used = 0;
+
+	c = buf[used++];
+	*type = (c >> 4) & 7;
+	size = c & 15;
+	shift = 4;
+	while (c & 0x80) {
+		if (len <= used || bitsizeof(long) <= shift) {
+			error("bad object header");
+			size = used = 0;
+			break;
+		}
+		c = buf[used++];
+		size += (c & 0x7f) << shift;
+		shift += 7;
+	}
+	*sizep = size;
+	return used;
+}
packfile.h +2 -0
index 1cfda1d00..9f36e0112 100644
@@ -62,6 +62,8 @@ extern void close_all_packs(void);
 extern void unuse_pack(struct pack_window **);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
+extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
+
 extern void release_pack_memory(size_t);
 
 extern int open_packed_git(struct packed_git *p);
sha1_file.c +0 -25
index 51bb4d1db..b999957b0 100644
@@ -914,31 +914,6 @@ void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 	return map_sha1_file_1(NULL, sha1, size);
 }
 
-unsigned long unpack_object_header_buffer(const unsigned char *buf,
-		unsigned long len, enum object_type *type, unsigned long *sizep)
-{
-	unsigned shift;
-	unsigned long size, c;
-	unsigned long used = 0;
-
-	c = buf[used++];
-	*type = (c >> 4) & 7;
-	size = c & 15;
-	shift = 4;
-	while (c & 0x80) {
-		if (len <= used || bitsizeof(long) <= shift) {
-			error("bad object header");
-			size = used = 0;
-			break;
-		}
-		c = buf[used++];
-		size += (c & 0x7f) << shift;
-		shift += 7;
-	}
-	*sizep = size;
-	return used;
-}
-
 static int unpack_sha1_short_header(git_zstream *stream,
 				    unsigned char *map, unsigned long mapsize,
 				    void *buffer, unsigned long bufsiz)
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#73 lore raw

PATCH v3 13/23 pack: move get_size_from_delta()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 packfile.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 packfile.h  |  1 +
 sha1_file.c | 39 ---------------------------------------
 4 files changed, 41 insertions(+), 40 deletions(-)
cache.h +0 -1
index 75cc0c497..87f65aeea 100644
@@ -1669,7 +1669,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)
 
 extern int is_pack_valid(struct packed_git *);
 extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
-extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
 extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
 
 /*
packfile.c +40 -0
index 43b708812..fa90b643e 100644
@@ -4,6 +4,7 @@
 #include "dir.h"
 #include "mergesort.h"
 #include "packfile.h"
+#include "delta.h"
 
 char *odb_pack_name(struct strbuf *buf,
 		    const unsigned char *sha1,
@@ -909,3 +910,42 @@ unsigned long unpack_object_header_buffer(const unsigned char *buf,
 	*sizep = size;
 	return used;
 }
+
+unsigned long get_size_from_delta(struct packed_git *p,
+				  struct pack_window **w_curs,
+			          off_t curpos)
+{
+	const unsigned char *data;
+	unsigned char delta_head[20], *in;
+	git_zstream stream;
+	int st;
+
+	memset(&stream, 0, sizeof(stream));
+	stream.next_out = delta_head;
+	stream.avail_out = sizeof(delta_head);
+
+	git_inflate_init(&stream);
+	do {
+		in = use_pack(p, w_curs, curpos, &stream.avail_in);
+		stream.next_in = in;
+		st = git_inflate(&stream, Z_FINISH);
+		curpos += stream.next_in - in;
+	} while ((st == Z_OK || st == Z_BUF_ERROR) &&
+		 stream.total_out < sizeof(delta_head));
+	git_inflate_end(&stream);
+	if ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head)) {
+		error("delta data unpack-initial failed");
+		return 0;
+	}
+
+	/* Examine the initial part of the delta to figure out
+	 * the result size.
+	 */
+	data = delta_head;
+
+	/* ignore base size */
+	get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
+
+	/* Read the result size */
+	return get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
+}
packfile.h +1 -0
index 9f36e0112..9c3bce6b2 100644
@@ -63,6 +63,7 @@ extern void unuse_pack(struct pack_window **);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
 extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
+extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
 
 extern void release_pack_memory(size_t);
 
sha1_file.c +0 -39
index b999957b0..5d016ad6b 100644
@@ -1100,45 +1100,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-unsigned long get_size_from_delta(struct packed_git *p,
-				  struct pack_window **w_curs,
-			          off_t curpos)
-{
-	const unsigned char *data;
-	unsigned char delta_head[20], *in;
-	git_zstream stream;
-	int st;
-
-	memset(&stream, 0, sizeof(stream));
-	stream.next_out = delta_head;
-	stream.avail_out = sizeof(delta_head);
-
-	git_inflate_init(&stream);
-	do {
-		in = use_pack(p, w_curs, curpos, &stream.avail_in);
-		stream.next_in = in;
-		st = git_inflate(&stream, Z_FINISH);
-		curpos += stream.next_in - in;
-	} while ((st == Z_OK || st == Z_BUF_ERROR) &&
-		 stream.total_out < sizeof(delta_head));
-	git_inflate_end(&stream);
-	if ((st != Z_STREAM_END) && stream.total_out != sizeof(delta_head)) {
-		error("delta data unpack-initial failed");
-		return 0;
-	}
-
-	/* Examine the initial part of the delta to figure out
-	 * the result size.
-	 */
-	data = delta_head;
-
-	/* ignore base size */
-	get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
-
-	/* Read the result size */
-	return get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
-}
-
 static off_t get_delta_base(struct packed_git *p,
 				    struct pack_window **w_curs,
 				    off_t *curpos,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#74 lore raw

PATCH v3 14/23 pack: move unpack_object_header()

In reply to Jonathan Tan
---
 cache.h     |  1 -
 packfile.c  | 26 ++++++++++++++++++++++++++
 packfile.h  |  1 +
 sha1_file.c | 26 --------------------------
 4 files changed, 27 insertions(+), 27 deletions(-)
cache.h +0 -1
index 87f65aeea..7adbc587d 100644
@@ -1669,7 +1669,6 @@ extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *)
 
 extern int is_pack_valid(struct packed_git *);
 extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
-extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
 
 /*
  * Iterate over the files in the loose-object parts of the object
packfile.c +26 -0
index fa90b643e..3543b37b8 100644
@@ -949,3 +949,29 @@ unsigned long get_size_from_delta(struct packed_git *p,
 	/* Read the result size */
 	return get_delta_hdr_size(&data, delta_head+sizeof(delta_head));
 }
+
+int unpack_object_header(struct packed_git *p,
+			 struct pack_window **w_curs,
+			 off_t *curpos,
+			 unsigned long *sizep)
+{
+	unsigned char *base;
+	size_t left;
+	size_t used;
+	enum object_type type;
+
+	/* use_pack() assures us we have [base, base + 20) available
+	 * as a range that we can look at.  (Its actually the hash
+	 * size that is assured.)  With our object header encoding
+	 * the maximum deflated object size is 2^137, which is just
+	 * insane, so we know won't exceed what we have been given.
+	 */
+	base = use_pack(p, w_curs, *curpos, &left);
+	used = unpack_object_header_buffer(base, left, &type, sizep);
+	if (!used) {
+		type = OBJ_BAD;
+	} else
+		*curpos += used;
+
+	return type;
+}
packfile.h +1 -0
index 9c3bce6b2..d22a528b5 100644
@@ -64,6 +64,7 @@ extern struct packed_git *add_packed_git(const char *path, size_t path_len, int
 
 extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
 extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
+extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
 
 extern void release_pack_memory(size_t);
 
sha1_file.c +0 -26
index 5d016ad6b..681dcf1c0 100644
@@ -1171,32 +1171,6 @@ static const unsigned char *get_delta_base_sha1(struct packed_git *p,
 		return NULL;
 }
 
-int unpack_object_header(struct packed_git *p,
-			 struct pack_window **w_curs,
-			 off_t *curpos,
-			 unsigned long *sizep)
-{
-	unsigned char *base;
-	size_t left;
-	size_t used;
-	enum object_type type;
-
-	/* use_pack() assures us we have [base, base + 20) available
-	 * as a range that we can look at.  (Its actually the hash
-	 * size that is assured.)  With our object header encoding
-	 * the maximum deflated object size is 2^137, which is just
-	 * insane, so we know won't exceed what we have been given.
-	 */
-	base = use_pack(p, w_curs, *curpos, &left);
-	used = unpack_object_header_buffer(base, left, &type, sizep);
-	if (!used) {
-		type = OBJ_BAD;
-	} else
-		*curpos += used;
-
-	return type;
-}
-
 static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)
 {
 	int type;
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#75 lore raw

PATCH v3 15/23 pack: move clear_delta_base_cache(), packed_object_info(), unpack_entry()

In reply to Jonathan Tan

Both sha1_file.c and packfile.c now need read_object(), so a copy of read_object() was created in packfile.c.

This patch makes both mark_bad_packed_object() and has_packed_and_bad() global. Unlike most of the other patches in this series, these 2 functions need to remain global.

---
 cache.h     |   7 -
 packfile.c  | 661 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 packfile.h  |  10 +
 sha1_file.c | 677 ++----------------------------------------------------------
 4 files changed, 685 insertions(+), 670 deletions(-)
cache.h +0 -7
index 7adbc587d..11aa18e6a 100644
@@ -1194,9 +1194,6 @@ extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
-/* global flag to enable extra checks when accessing packed objects */
-extern int do_check_packed_object_crc;
-
 extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned long size, const char *type);
 
 extern int finalize_object_file(const char *tmpfile, const char *filename);
@@ -1629,8 +1626,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-extern void clear_delta_base_cache(void);
-
 /*
  * Make sure that a pointer access into an mmap'd index file is within bounds,
  * and can provide at least 8 bytes of data.
@@ -1668,7 +1663,6 @@ extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
 extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);
 
 extern int is_pack_valid(struct packed_git *);
-extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
 
 /*
  * Iterate over the files in the loose-object parts of the object
@@ -1779,7 +1773,6 @@ struct object_info {
 /* Do not retry packed storage after checking packed and loose storage */
 #define OBJECT_INFO_QUICK 8
 extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);
-extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);
 
 /* Dumb servers support */
 extern int update_server_info(int);
packfile.c +661 -0

670 lines not on this page · raw message

packfile.h +10 -0
index d22a528b5..c28eaccc6 100644
@@ -60,8 +60,10 @@ extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t
 extern void close_pack_windows(struct packed_git *);
 extern void close_all_packs(void);
 extern void unuse_pack(struct pack_window **);
+extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
+extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
 extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
 extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
 extern int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *);
@@ -70,4 +72,12 @@ extern void release_pack_memory(size_t);
 
 extern int open_packed_git(struct packed_git *p);
 
+/* global flag to enable extra checks when accessing packed objects */
+extern int do_check_packed_object_crc;
+
+extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);
+
+extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);
+extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);
+
 #endif
sha1_file.c +14 -663

701 lines not on this page · raw message

Signature · 1 line
2.14.1.480.gb18f417b89-goog
#76 lore raw

PATCH v3 16/23 pack: move nth_packed_object_{sha1,oid}

In reply to Jonathan Tan
---
 cache.h     | 14 --------------
 packfile.c  | 31 +++++++++++++++++++++++++++++++
 packfile.h  | 16 +++++++++++++++-
 sha1_file.c | 31 -------------------------------
 4 files changed, 46 insertions(+), 46 deletions(-)
cache.h +0 -14
index 11aa18e6a..83aa3cc62 100644
@@ -1636,20 +1636,6 @@ extern int odb_pack_keep(const char *name);
  */
 extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);
 
-/*
- * Return the SHA-1 of the nth object within the specified packfile.
- * Open the index if it is not already open.  The return value points
- * at the SHA-1 within the mmapped index.  Return NULL if there is an
- * error.
- */
-extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n);
-/*
- * Like nth_packed_object_sha1, but write the data into the object specified by
- * the the first argument.  Returns the first argument on success, and NULL on
- * error.
- */
-extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);
-
 /*
  * Return the offset of the nth object within the specified packfile.
  * The index must already be opened.
packfile.c +31 -0
index 624cc109e..e9b16da94 100644
@@ -1636,3 +1636,34 @@ void *unpack_entry(struct packed_git *p, off_t obj_offset,
 
 	return data;
 }
+
+const unsigned char *nth_packed_object_sha1(struct packed_git *p,
+					    uint32_t n)
+{
+	const unsigned char *index = p->index_data;
+	if (!index) {
+		if (open_pack_index(p))
+			return NULL;
+		index = p->index_data;
+	}
+	if (n >= p->num_objects)
+		return NULL;
+	index += 4 * 256;
+	if (p->index_version == 1) {
+		return index + 24 * n + 4;
+	} else {
+		index += 8;
+		return index + 20 * n;
+	}
+}
+
+const struct object_id *nth_packed_object_oid(struct object_id *oid,
+					      struct packed_git *p,
+					      uint32_t n)
+{
+	const unsigned char *hash = nth_packed_object_sha1(p, n);
+	if (!hash)
+		return NULL;
+	hashcpy(oid->hash, hash);
+	return oid;
+}
packfile.h +15 -1
index c28eaccc6..56d70caa0 100644
@@ -63,6 +63,21 @@ extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
+/*
+ * Return the SHA-1 of the nth object within the specified packfile.
+ * Open the index if it is not already open.  The return value points
+ * at the SHA-1 within the mmapped index.  Return NULL if there is an
+ * error.
+ */
+extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t n);
+/*
+ * Like nth_packed_object_sha1, but write the data into the object specified by
+ * the the first argument.  Returns the first argument on success, and NULL on
+ * error.
+ */
+extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);
+
+
 extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
 extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
 extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
@@ -79,5 +94,4 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct obje
 
 extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);
 extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);
-
 #endif
sha1_file.c +0 -31
index e537ba089..34fbe8e51 100644
@@ -1074,37 +1074,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-const unsigned char *nth_packed_object_sha1(struct packed_git *p,
-					    uint32_t n)
-{
-	const unsigned char *index = p->index_data;
-	if (!index) {
-		if (open_pack_index(p))
-			return NULL;
-		index = p->index_data;
-	}
-	if (n >= p->num_objects)
-		return NULL;
-	index += 4 * 256;
-	if (p->index_version == 1) {
-		return index + 24 * n + 4;
-	} else {
-		index += 8;
-		return index + 20 * n;
-	}
-}
-
-const struct object_id *nth_packed_object_oid(struct object_id *oid,
-					      struct packed_git *p,
-					      uint32_t n)
-{
-	const unsigned char *hash = nth_packed_object_sha1(p, n);
-	if (!hash)
-		return NULL;
-	hashcpy(oid->hash, hash);
-	return oid;
-}
-
 void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
 {
 	const unsigned char *ptr = vptr;
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#77 lore raw

PATCH v3 17/23 pack: move check_pack_index_ptr(), nth_packed_object_offset()

In reply to Jonathan Tan
---
 cache.h     | 16 ----------------
 packfile.c  | 33 +++++++++++++++++++++++++++++++++
 packfile.h  | 16 ++++++++++++++++
 sha1_file.c | 33 ---------------------------------
 4 files changed, 49 insertions(+), 49 deletions(-)
cache.h +0 -16
index 83aa3cc62..ee75a4949 100644
@@ -1626,22 +1626,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * Make sure that a pointer access into an mmap'd index file is within bounds,
- * and can provide at least 8 bytes of data.
- *
- * Note that this is only necessary for variable-length segments of the file
- * (like the 64-bit extended offset table), as we compare the size to the
- * fixed-length parts when we open the file.
- */
-extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);
-
-/*
- * Return the offset of the nth object within the specified packfile.
- * The index must already be opened.
- */
-extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
-
 /*
  * If the object named sha1 is present in the specified packfile,
  * return its offset within the packfile; otherwise, return 0.
packfile.c +33 -0
index e9b16da94..e914422e9 100644
@@ -1667,3 +1667,36 @@ const struct object_id *nth_packed_object_oid(struct object_id *oid,
 	hashcpy(oid->hash, hash);
 	return oid;
 }
+
+void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
+{
+	const unsigned char *ptr = vptr;
+	const unsigned char *start = p->index_data;
+	const unsigned char *end = start + p->index_size;
+	if (ptr < start)
+		die(_("offset before start of pack index for %s (corrupt index?)"),
+		    p->pack_name);
+	/* No need to check for underflow; .idx files must be at least 8 bytes */
+	if (ptr >= end - 8)
+		die(_("offset beyond end of pack index for %s (truncated index?)"),
+		    p->pack_name);
+}
+
+off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
+{
+	const unsigned char *index = p->index_data;
+	index += 4 * 256;
+	if (p->index_version == 1) {
+		return ntohl(*((uint32_t *)(index + 24 * n)));
+	} else {
+		uint32_t off;
+		index += 8 + p->num_objects * (20 + 4);
+		off = ntohl(*((uint32_t *)(index + 4 * n)));
+		if (!(off & 0x80000000))
+			return off;
+		index += p->num_objects * 4 + (off & 0x7fffffff) * 8;
+		check_pack_index_ptr(p, index);
+		return (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |
+				   ntohl(*((uint32_t *)(index + 4)));
+	}
+}
packfile.h +16 -0
index 56d70caa0..8deb84bd1 100644
@@ -63,6 +63,16 @@ extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
 
+/*
+ * Make sure that a pointer access into an mmap'd index file is within bounds,
+ * and can provide at least 8 bytes of data.
+ *
+ * Note that this is only necessary for variable-length segments of the file
+ * (like the 64-bit extended offset table), as we compare the size to the
+ * fixed-length parts when we open the file.
+ */
+extern void check_pack_index_ptr(const struct packed_git *p, const void *ptr);
+
 /*
  * Return the SHA-1 of the nth object within the specified packfile.
  * Open the index if it is not already open.  The return value points
@@ -77,6 +87,11 @@ extern const unsigned char *nth_packed_object_sha1(struct packed_git *, uint32_t
  */
 extern const struct object_id *nth_packed_object_oid(struct object_id *, struct packed_git *, uint32_t n);
 
+/*
+ * Return the offset of the nth object within the specified packfile.
+ * The index must already be opened.
+ */
+extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
 
 extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
 extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
@@ -94,4 +109,5 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct obje
 
 extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);
 extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);
+
 #endif
sha1_file.c +0 -33
index 34fbe8e51..2d22bc228 100644
@@ -1074,39 +1074,6 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-void check_pack_index_ptr(const struct packed_git *p, const void *vptr)
-{
-	const unsigned char *ptr = vptr;
-	const unsigned char *start = p->index_data;
-	const unsigned char *end = start + p->index_size;
-	if (ptr < start)
-		die(_("offset before start of pack index for %s (corrupt index?)"),
-		    p->pack_name);
-	/* No need to check for underflow; .idx files must be at least 8 bytes */
-	if (ptr >= end - 8)
-		die(_("offset beyond end of pack index for %s (truncated index?)"),
-		    p->pack_name);
-}
-
-off_t nth_packed_object_offset(const struct packed_git *p, uint32_t n)
-{
-	const unsigned char *index = p->index_data;
-	index += 4 * 256;
-	if (p->index_version == 1) {
-		return ntohl(*((uint32_t *)(index + 24 * n)));
-	} else {
-		uint32_t off;
-		index += 8 + p->num_objects * (20 + 4);
-		off = ntohl(*((uint32_t *)(index + 4 * n)));
-		if (!(off & 0x80000000))
-			return off;
-		index += p->num_objects * 4 + (off & 0x7fffffff) * 8;
-		check_pack_index_ptr(p, index);
-		return (((uint64_t)ntohl(*((uint32_t *)(index + 0)))) << 32) |
-				   ntohl(*((uint32_t *)(index + 4)));
-	}
-}
-
 off_t find_pack_entry_one(const unsigned char *sha1,
 				  struct packed_git *p)
 {
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#78 lore raw

PATCH v3 18/23 pack: move find_pack_entry_one(), is_pack_valid()

In reply to Jonathan Tan
---
 cache.h     |  8 -------
 packfile.c  | 76 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 packfile.h  |  9 ++++++--
 sha1_file.c | 73 ----------------------------------------------------------
 4 files changed, 82 insertions(+), 84 deletions(-)
cache.h +0 -8
index ee75a4949..9297d078a 100644
@@ -1626,14 +1626,6 @@ extern int odb_mkstemp(struct strbuf *template, const char *pattern);
  */
 extern int odb_pack_keep(const char *name);
 
-/*
- * If the object named sha1 is present in the specified packfile,
- * return its offset within the packfile; otherwise, return 0.
- */
-extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);
-
-extern int is_pack_valid(struct packed_git *);
-
 /*
  * Iterate over the files in the loose-object parts of the object
  * directory "path", triggering the following callbacks:
packfile.c +75 -1

91 lines not on this page · raw message

packfile.h +7 -2
index 8deb84bd1..4fca6fb28 100644
@@ -93,6 +93,13 @@ extern const struct object_id *nth_packed_object_oid(struct object_id *, struct
  */
 extern off_t nth_packed_object_offset(const struct packed_git *, uint32_t n);
 
+/*
+ * If the object named sha1 is present in the specified packfile,
+ * return its offset within the packfile; otherwise, return 0.
+ */
+extern off_t find_pack_entry_one(const unsigned char *sha1, struct packed_git *);
+
+extern int is_pack_valid(struct packed_git *);
 extern void *unpack_entry(struct packed_git *, off_t, enum object_type *, unsigned long *);
 extern unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep);
 extern unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t);
@@ -100,8 +107,6 @@ extern int unpack_object_header(struct packed_git *, struct pack_window **, off_
 
 extern void release_pack_memory(size_t);
 
-extern int open_packed_git(struct packed_git *p);
-
 /* global flag to enable extra checks when accessing packed objects */
 extern int do_check_packed_object_crc;
 
sha1_file.c +0 -73

79 lines not on this page · raw message

Signature · 1 line
2.14.1.480.gb18f417b89-goog
#79 lore raw

PATCH v3 19/23 pack: move find_sha1_pack()

In reply to Jonathan Tan
---
 cache.h       |  3 ---
 http-push.c   |  1 +
 http-walker.c |  1 +
 packfile.c    | 13 +++++++++++++
 packfile.h    |  3 +++
 sha1_file.c   | 13 -------------
 6 files changed, 18 insertions(+), 16 deletions(-)
cache.h +0 -3
index 9297d078a..1e90bb754 100644
@@ -1608,9 +1608,6 @@ struct pack_entry {
 	struct packed_git *p;
 };
 
-extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
-					 struct packed_git *packs);
-
 /*
  * Create a temporary file rooted in the object database directory, or
  * die on failure. The filename is taken from "pattern", which should have the
http-push.c +1 -0
index c91f40a61..e4c9b065c 100644
@@ -11,6 +11,7 @@
 #include "list-objects.h"
 #include "sigchain.h"
 #include "argv-array.h"
+#include "packfile.h"
 
 #ifdef EXPAT_NEEDS_XMLPARSE_H
 #include <xmlparse.h>
http-walker.c +1 -0
index ee049cb13..1ae8363de 100644
@@ -4,6 +4,7 @@
 #include "http.h"
 #include "list.h"
 #include "transport.h"
+#include "packfile.h"
 
 struct alt_base {
 	char *base;
packfile.c +13 -0
index ad7336594..ba3a5eb3a 100644
@@ -1774,3 +1774,16 @@ int is_pack_valid(struct packed_git *p)
 	/* Force the pack to open to prove its valid. */
 	return !open_packed_git(p);
 }
+
+struct packed_git *find_sha1_pack(const unsigned char *sha1,
+				  struct packed_git *packs)
+{
+	struct packed_git *p;
+
+	for (p = packs; p; p = p->next) {
+		if (find_pack_entry_one(sha1, p))
+			return p;
+	}
+	return NULL;
+
+}
packfile.h +3 -0
index 4fca6fb28..a4ff6f6ed 100644
@@ -42,6 +42,9 @@ extern void install_packed_git(struct packed_git *pack);
  */
 unsigned long approximate_object_count(void);
 
+extern struct packed_git *find_sha1_pack(const unsigned char *sha1,
+					 struct packed_git *packs);
+
 extern void pack_report(void);
 
 /*
sha1_file.c +0 -13
index 27714f5e1..8853672d2 100644
@@ -1127,19 +1127,6 @@ static int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	return 0;
 }
 
-struct packed_git *find_sha1_pack(const unsigned char *sha1,
-				  struct packed_git *packs)
-{
-	struct packed_git *p;
-
-	for (p = packs; p; p = p->next) {
-		if (find_pack_entry_one(sha1, p))
-			return p;
-	}
-	return NULL;
-
-}
-
 static int sha1_loose_object_info(const unsigned char *sha1,
 				  struct object_info *oi,
 				  int flags)
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#80 lore raw

PATCH v3 20/23 pack: move find_pack_entry() and make it global

In reply to Jonathan Tan

This function needs to be global as it is used by sha1_file.c and will be used by packfile.c.

---
 packfile.c  | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 packfile.h  |  2 ++
 sha1_file.c | 53 -----------------------------------------------------
 3 files changed, 55 insertions(+), 53 deletions(-)
packfile.c +53 -0
index ba3a5eb3a..ae5395f5f 100644
@@ -1787,3 +1787,56 @@ struct packed_git *find_sha1_pack(const unsigned char *sha1,
 	return NULL;
 
 }
+
+static int fill_pack_entry(const unsigned char *sha1,
+			   struct pack_entry *e,
+			   struct packed_git *p)
+{
+	off_t offset;
+
+	if (p->num_bad_objects) {
+		unsigned i;
+		for (i = 0; i < p->num_bad_objects; i++)
+			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
+				return 0;
+	}
+
+	offset = find_pack_entry_one(sha1, p);
+	if (!offset)
+		return 0;
+
+	/*
+	 * We are about to tell the caller where they can locate the
+	 * requested object.  We better make sure the packfile is
+	 * still here and can be accessed before supplying that
+	 * answer, as it may have been deleted since the index was
+	 * loaded!
+	 */
+	if (!is_pack_valid(p))
+		return 0;
+	e->offset = offset;
+	e->p = p;
+	hashcpy(e->sha1, sha1);
+	return 1;
+}
+
+/*
+ * Iff a pack file contains the object named by sha1, return true and
+ * store its location to e.
+ */
+int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
+{
+	struct mru_entry *p;
+
+	prepare_packed_git();
+	if (!packed_git)
+		return 0;
+
+	for (p = packed_git_mru->head; p; p = p->next) {
+		if (fill_pack_entry(sha1, e, p->item)) {
+			mru_mark(packed_git_mru, p);
+			return 1;
+		}
+	}
+	return 0;
+}
packfile.h +2 -0
index a4ff6f6ed..c9b4fcfaf 100644
@@ -118,4 +118,6 @@ extern int packed_object_info(struct packed_git *pack, off_t offset, struct obje
 extern void mark_bad_packed_object(struct packed_git *p, const unsigned char *sha1);
 extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);
 
+extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);
+
 #endif
sha1_file.c +0 -53

59 lines not on this page · raw message

Signature · 1 line
2.14.1.480.gb18f417b89-goog
#81 lore raw

PATCH v3 21/23 pack: move has_sha1_pack()

In reply to Jonathan Tan
---
 builtin/prune-packed.c | 1 +
 cache.h                | 2 --
 diff.c                 | 1 +
 packfile.c             | 6 ++++++
 packfile.h             | 2 ++
 revision.c             | 1 +
 sha1_file.c            | 6 ------
 7 files changed, 11 insertions(+), 8 deletions(-)
builtin/prune-packed.c +1 -0
index ac978ad40..97bfde24b 100644
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "progress.h"
 #include "parse-options.h"
+#include "packfile.h"
 
 static const char * const prune_packed_usage[] = {
 	N_("git prune-packed [-n | --dry-run] [-q | --quiet]"),
cache.h +0 -2
index 1e90bb754..286891df4 100644
@@ -1198,8 +1198,6 @@ extern int check_sha1_signature(const unsigned char *sha1, void *buf, unsigned l
 
 extern int finalize_object_file(const char *tmpfile, const char *filename);
 
-extern int has_sha1_pack(const unsigned char *sha1);
-
 /*
  * Open the loose object at path, check its sha1, and return the contents,
  * type, and size. If the object is a blob, then "contents" may return NULL,
diff.c +1 -0
index 85e714f6c..e9a1d6162 100644
@@ -20,6 +20,7 @@
 #include "string-list.h"
 #include "argv-array.h"
 #include "graph.h"
+#include "packfile.h"
 
 #ifdef NO_FAST_WORKING_DIRECTORY
 #define FAST_WORKING_DIRECTORY 0
packfile.c +6 -0
index ae5395f5f..7472ab816 100644
@@ -1840,3 +1840,9 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	}
 	return 0;
 }
+
+int has_sha1_pack(const unsigned char *sha1)
+{
+	struct pack_entry e;
+	return find_pack_entry(sha1, &e);
+}
packfile.h +2 -0
index c9b4fcfaf..4945a1505 100644
@@ -120,4 +120,6 @@ extern const struct packed_git *has_packed_and_bad(const unsigned char *sha1);
 
 extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);
 
+extern int has_sha1_pack(const unsigned char *sha1);
+
 #endif
revision.c +1 -0
index 6603af944..db97b9221 100644
@@ -19,6 +19,7 @@
 #include "dir.h"
 #include "cache-tree.h"
 #include "bisect.h"
+#include "packfile.h"
 
 volatile show_early_output_fn_t show_early_output;
 
sha1_file.c +0 -6
index 76c86639c..e4975e0ae 100644
@@ -1630,12 +1630,6 @@ int has_pack_index(const unsigned char *sha1)
 	return 1;
 }
 
-int has_sha1_pack(const unsigned char *sha1)
-{
-	struct pack_entry e;
-	return find_pack_entry(sha1, &e);
-}
-
 int has_sha1_file_with_flags(const unsigned char *sha1, int flags)
 {
 	if (!startup_info->have_repository)
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#82 lore raw

PATCH v3 22/23 pack: move has_pack_index()

In reply to Jonathan Tan
---
 cache.h     | 2 --
 packfile.c  | 8 ++++++++
 packfile.h  | 2 ++
 sha1_file.c | 8 --------
 4 files changed, 10 insertions(+), 10 deletions(-)
cache.h +0 -2
index 286891df4..dcbe37a3f 100644
@@ -1233,8 +1233,6 @@ extern int has_object_file_with_flags(const struct object_id *oid, int flags);
  */
 extern int has_loose_object_nonlocal(const unsigned char *sha1);
 
-extern int has_pack_index(const unsigned char *sha1);
-
 extern void assert_sha1_type(const unsigned char *sha1, enum object_type expect);
 
 /* Helper to check and "touch" a file */
packfile.c +8 -0
index 7472ab816..7e293761b 100644
@@ -1846,3 +1846,11 @@ int has_sha1_pack(const unsigned char *sha1)
 	struct pack_entry e;
 	return find_pack_entry(sha1, &e);
 }
+
+int has_pack_index(const unsigned char *sha1)
+{
+	struct stat st;
+	if (stat(sha1_pack_index_name(sha1), &st))
+		return 0;
+	return 1;
+}
packfile.h +2 -0
index 4945a1505..1b6ea832c 100644
@@ -122,4 +122,6 @@ extern int find_pack_entry(const unsigned char *sha1, struct pack_entry *e);
 
 extern int has_sha1_pack(const unsigned char *sha1);
 
+extern int has_pack_index(const unsigned char *sha1);
+
 #endif
sha1_file.c +0 -8
index e4975e0ae..fa422435f 100644
@@ -1622,14 +1622,6 @@ int force_object_loose(const unsigned char *sha1, time_t mtime)
 	return ret;
 }
 
-int has_pack_index(const unsigned char *sha1)
-{
-	struct stat st;
-	if (stat(sha1_pack_index_name(sha1), &st))
-		return 0;
-	return 1;
-}
-
 int has_sha1_file_with_flags(const unsigned char *sha1, int flags)
 {
 	if (!startup_info->have_repository)
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#83 lore raw

PATCH v3 23/23 pack: move for_each_packed_object()

In reply to Jonathan Tan
---
 builtin/cat-file.c |  1 +
 cache.h            |  7 +------
 packfile.c         | 40 ++++++++++++++++++++++++++++++++++++++++
 packfile.h         | 11 +++++++++++
 reachable.c        |  1 +
 sha1_file.c        | 40 ----------------------------------------
 6 files changed, 54 insertions(+), 46 deletions(-)
builtin/cat-file.c +1 -0
index 96b786e48..be5936017 100644
@@ -12,6 +12,7 @@
 #include "streaming.h"
 #include "tree-walk.h"
 #include "sha1-array.h"
+#include "packfile.h"
 
 struct batch_options {
 	int enabled;
cache.h +1 -6
index dcbe37a3f..2eeb21b02 100644
@@ -1668,17 +1668,12 @@ int for_each_loose_file_in_objdir_buf(struct strbuf *path,
 				      void *data);
 
 /*
- * Iterate over loose and packed objects in both the local
+ * Iterate over loose objects in both the local
  * repository and any alternates repositories (unless the
  * LOCAL_ONLY flag is set).
  */
 #define FOR_EACH_OBJECT_LOCAL_ONLY 0x1
-typedef int each_packed_object_fn(const struct object_id *oid,
-				  struct packed_git *pack,
-				  uint32_t pos,
-				  void *data);
 extern int for_each_loose_object(each_loose_object_fn, void *, unsigned flags);
-extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);
 
 struct object_info {
 	/* Request */
packfile.c +40 -0
index 7e293761b..1f11ef5b8 100644
@@ -1854,3 +1854,43 @@ int has_pack_index(const unsigned char *sha1)
 		return 0;
 	return 1;
 }
+
+static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)
+{
+	uint32_t i;
+	int r = 0;
+
+	for (i = 0; i < p->num_objects; i++) {
+		struct object_id oid;
+
+		if (!nth_packed_object_oid(&oid, p, i))
+			return error("unable to get sha1 of object %u in %s",
+				     i, p->pack_name);
+
+		r = cb(&oid, p, i, data);
+		if (r)
+			break;
+	}
+	return r;
+}
+
+int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
+{
+	struct packed_git *p;
+	int r = 0;
+	int pack_errors = 0;
+
+	prepare_packed_git();
+	for (p = packed_git; p; p = p->next) {
+		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
+			continue;
+		if (open_pack_index(p)) {
+			pack_errors = 1;
+			continue;
+		}
+		r = for_each_object_in_pack(p, cb, data);
+		if (r)
+			break;
+	}
+	return r ? r : pack_errors;
+}
packfile.h +11 -0
index 1b6ea832c..ca4cc3b97 100644
@@ -124,4 +124,15 @@ extern int has_sha1_pack(const unsigned char *sha1);
 
 extern int has_pack_index(const unsigned char *sha1);
 
+/*
+ * Iterate over packed objects in both the local
+ * repository and any alternates repositories (unless the
+ * FOR_EACH_OBJECT_LOCAL_ONLY flag, defined in cache.h, is set).
+ */
+typedef int each_packed_object_fn(const struct object_id *oid,
+				  struct packed_git *pack,
+				  uint32_t pos,
+				  void *data);
+extern int for_each_packed_object(each_packed_object_fn, void *, unsigned flags);
+
 #endif
reachable.c +1 -0
index c62efbfd4..d1ac5d97e 100644
@@ -9,6 +9,7 @@
 #include "cache-tree.h"
 #include "progress.h"
 #include "list-objects.h"
+#include "packfile.h"
 
 struct connectivity_progress {
 	struct progress *progress;
sha1_file.c +0 -40
index fa422435f..0bb2343f8 100644
@@ -2014,46 +2014,6 @@ int for_each_loose_object(each_loose_object_fn cb, void *data, unsigned flags)
 	return foreach_alt_odb(loose_from_alt_odb, &alt);
 }
 
-static int for_each_object_in_pack(struct packed_git *p, each_packed_object_fn cb, void *data)
-{
-	uint32_t i;
-	int r = 0;
-
-	for (i = 0; i < p->num_objects; i++) {
-		struct object_id oid;
-
-		if (!nth_packed_object_oid(&oid, p, i))
-			return error("unable to get sha1 of object %u in %s",
-				     i, p->pack_name);
-
-		r = cb(&oid, p, i, data);
-		if (r)
-			break;
-	}
-	return r;
-}
-
-int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
-{
-	struct packed_git *p;
-	int r = 0;
-	int pack_errors = 0;
-
-	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
-		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
-			continue;
-		if (open_pack_index(p)) {
-			pack_errors = 1;
-			continue;
-		}
-		r = for_each_object_in_pack(p, cb, data);
-		if (r)
-			break;
-	}
-	return r ? r : pack_errors;
-}
-
 static int check_stream_sha1(git_zstream *stream,
 			     const char *hdr,
 			     unsigned long size,
Signature · 1 line
2.14.1.480.gb18f417b89-goog
#84 lore raw

Re: [PATCH v3 00/23] Move exported packfile funcs to its own file

In reply to Jonathan Tan
Quoted 26 lines · Jonathan Tan
Quoted 8 lines

You'd need to double check, but I think the topics that cause trouble are rs/find-apck-entry-bisection and jk/drop-sha1-entry-pos; you can start from v2.14.1 and merge these topics on top and then build your change on top. That would allow you to start cooking before both of them graduate to 'master', as I expect they are both quick-to-next material. There might be other topics that interfere with what you are doing, but you can easily find out what they are if you do a trial merge to 'next' and 'pu' yourself.

OK - in addition to the 2 you mentioned, I have found some others (likely added after you wrote that). The complete list is:
- rs/find-pack-entry-bisection
- jk/drop-sha1-entry-pos
- jt/sha1-file-cleanup (formerly part of this set)
- mk/use-size-t-in-zlib
- rs/unpack-entry-leakfix

I have merged all of these and rebased my patches on top.

Other changes:
- Used packfile.h instead of pack.h (following most people's preference)
- Ensured that I added functions to packfile.h retaining the order they were originally in, so that if you run "git diff <base> --color-moved --patience", there are much fewer zebra stripes

Quoted 3 lines

The merge base commit can be accessed online [1], if you need it.

[1] https://github.com/jonathantanmy/git/commits/packmigrate

Thanks.

I have to say that this was a painful topic to integrate.

As you may know, the mk/use-size-t-in-zlib topic is being retracted and getting rerolled as a larger size_t series, most of which still needs help in reviewing.

The jt/sha1-file-cleanup topic is the only one among the other four that are still not in 'next', and I think that topic, as well as the other three, are all good and serve as a good base to build on top. So I first rebuilt your patches on top of these four topics. This took some time but it wasn't all that painful.

The result cleanly merged to 'pu', I think, but it resulted in a rather noisy conflict when I attempted to merge it to 'next'. I want to see both of a merge to 'next' and a merge to 'pu' to be reasonably clean for any topic to be viable [*1*]. Otherwise, "initially queue in 'pu', then cook in 'next', and eventually graduate to 'master'" workflow would not work well.

Anyway, I _think_ I finally got the conflict resolutions right for merges of the topic to 'next', 'jch' and 'pu', so I will push the result of merging to 'pu' out. This unfortunately makes Martin's ongoing size_t topic unmergeable to any of the integration branches as-is, but let's make sure that topic is reviewed properly first (I haven't seen people comment much on the individual patches other than just selected few).

[Footnote]

*1* There is an intermediate point between 'master' and 'pu' called
'jch', and I try to make sure any new topic to merge cleanly to that branch, too, when accepting it. That is the branch I use for everyday work as an early guinea-pig.

#85 lore raw

Re: [PATCH v3 00/23] Move exported packfile funcs to its own file

Quoted 11 lines · Junio C Hamano

I have to say that this was a painful topic to integrate.

As you may know, the mk/use-size-t-in-zlib topic is being retracted and getting rerolled as a larger size_t series, most of which still needs help in reviewing.

The jt/sha1-file-cleanup topic is the only one among the other four that are still not in 'next', and I think that topic, as well as the other three, are all good and serve as a good base to build on top. So I first rebuilt your patches on top of these four topics. This took some time but it wasn't all that painful.

... but it turns out that I screwed it up in at least one place, making Linux32 build fail (Thanks Lars and folks who pushed hard to arrange Travis to build all my pushes to 'pu'). I'm pushing out my second attempt. Let's see how it goes.

A change like this that only moves code around without changing anything is painful to everybody to keep around, as nobody can safely touch the affected code while it is in flight. On the other hand, as long as it is only moving code around, such a change is reasonably safe, and it is relatively easy to ensure that there is no change other than code movement is involved. So let's

(1) make sure that the topics this depends on are sound by re-reading them once again, and merge them quickly down to 'master';

(2) merge this topic to 'next', optionally after rebasing it on 'master', after (1) happens; and

(3) quickly merge it to 'master', to get it over with.

In the meantime we'd need to refrain from taking code that touch things that are moved by this series.

I plan to be offline for a week or so near the end of this month, so I am hoping that we can do all of the above before that. That may make us break our usual "tip of the 'master' is more stable and robust than any released version" promise by potentially leaving it broken for a while, but nobody can build on top of a fluid codebase that is in the process of moving things around in a big way, so it might not be such a bad idea to make it coincide with the period when the tree must become quiescent due to my being offline. We'll see.

Thanks.

#86 lore raw

Re: [PATCH v3 00/23] Move exported packfile funcs to its own file

Quoted 14 lines · On Sat, 19 Aug 2017 23:40:33 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 11 lines · Junio C Hamano

I have to say that this was a painful topic to integrate.

As you may know, the mk/use-size-t-in-zlib topic is being retracted and getting rerolled as a larger size_t series, most of which still needs help in reviewing.

The jt/sha1-file-cleanup topic is the only one among the other four that are still not in 'next', and I think that topic, as well as the other three, are all good and serve as a good base to build on top. So I first rebuilt your patches on top of these four topics. This took some time but it wasn't all that painful.

Quoted 4 lines

... but it turns out that I screwed it up in at least one place, making Linux32 build fail (Thanks Lars and folks who pushed hard to arrange Travis to build all my pushes to 'pu'). I'm pushing out my second attempt. Let's see how it goes.

Thanks.

Quoted 10 lines

A change like this that only moves code around without changing anything is painful to everybody to keep around, as nobody can safely touch the affected code while it is in flight. On the other hand, as long as it is only moving code around, such a change is reasonably safe, and it is relatively easy to ensure that there is no change other than code movement is involved. So let's

(1) make sure that the topics this depends on are sound by re-reading them once again, and merge them quickly down to 'master';

I took a look and they look sound.

- rs/find-pack-entry-bisection resolves an issue first introduced in commit 1f68855 ("[PATCH] Teach read_sha1_file() and friends about packed git object store.", 2005-06-27), which already had that issue.
- jk/drop-sha1-entry-pos is some code deletion.
- rs/unpack-entry-leakfix ensures that delta_stack is freed. This function does not (for example) expose the destination of delta_stack to its caller, so it is correct that delta_stack should be freed unless it points to the local buffer, just like in the success case.
- jt/sha1-file-cleanup (my patches) still looks OK to me.

In your latest "What's cooking" (Aug 2017, #04; Fri, 18), you mentioned that the first 3 will be merged to master, and the 4th will be merged to next.

I didn't look at mk/use-size-t-in-zlib, which (as you said) is still under review.

Quoted 19 lines

(2) merge this topic to 'next', optionally after rebasing it on 'master', after (1) happens; and

(3) quickly merge it to 'master', to get it over with.

In the meantime we'd need to refrain from taking code that touch things that are moved by this series.

I plan to be offline for a week or so near the end of this month, so I am hoping that we can do all of the above before that. That may make us break our usual "tip of the 'master' is more stable and robust than any released version" promise by potentially leaving it broken for a while, but nobody can build on top of a fluid codebase that is in the process of moving things around in a big way, so it might not be such a bad idea to make it coincide with the period when the tree must become quiescent due to my being offline. We'll see.

Thanks.

#87 lore raw

Re: [PATCH v3 00/23] Move exported packfile funcs to its own file

Quoted 17 lines · Jonathan Tan
Quoted 14 lines · On Sat, 19 Aug 2017 23:40:33 -0700 Junio C Hamano <[email protected]> wrote:
Quoted 11 lines · Junio C Hamano

I have to say that this was a painful topic to integrate.

As you may know, the mk/use-size-t-in-zlib topic is being retracted and getting rerolled as a larger size_t series, most of which still needs help in reviewing.

The jt/sha1-file-cleanup topic is the only one among the other four that are still not in 'next', and I think that topic, as well as the other three, are all good and serve as a good base to build on top. So I first rebuilt your patches on top of these four topics. This took some time but it wasn't all that painful.

Quoted 6 lines
Quoted 4 lines

... but it turns out that I screwed it up in at least one place, making Linux32 build fail (Thanks Lars and folks who pushed hard to arrange Travis to build all my pushes to 'pu'). I'm pushing out my second attempt. Let's see how it goes.

Thanks.

It seems like a later pushout late Sunday night that had the second attempt made it pass on Linux32 ;-) Whew.

Quoted 16 lines
Quoted 3 lines

(1) make sure that the topics this depends on are sound by re-reading them once again, and merge them quickly down to 'master';

I took a look and they look sound.

- rs/find-pack-entry-bisection resolves an issue first introduced in commit 1f68855 ("[PATCH] Teach read_sha1_file() and friends about packed git object store.", 2005-06-27), which already had that issue.
- jk/drop-sha1-entry-pos is some code deletion.
- rs/unpack-entry-leakfix ensures that delta_stack is freed. This function does not (for example) expose the destination of delta_stack to its caller, so it is correct that delta_stack should be freed unless it points to the local buffer, just like in the success case.
- jt/sha1-file-cleanup (my patches) still looks OK to me.

Quoted 3 lines

In your latest "What's cooking" (Aug 2017, #04; Fri, 18), you mentioned that the first 3 will be merged to master, and the 4th will be merged to next.

Yup, thanks for double checking. I'll be merging them down soon-ish.

Thanks.