Skip to contentSkip to conversation

Don't lazy-fetch commits when parsing them

PATCH Superseded Jonathan Tan 4 participants 85 messages Nov 30, 2022 – Dec 15, 2022

lore
Share & tools
lore
#0 lore raw

cover Don't lazy-fetch commits when parsing them

This is a follow-up from my previous email about the possibility of not fetching when we know that we're fetching a commit [1]. I had to refactor a few things mostly due to replace objects, so the number of patches might be larger than you would expect. I tried to keep each patch small and easy to understand, though.

Patches 1-3 contain some forward-compatibility and refactoring changes, and patch 4 contains the actual logic change.

[1] https://lore.kernel.org/git/[email protected]/

Jonathan Tan (4):
object-file: reread object with exact same args
object-file: refactor corrupt object diagnosis
object-file: refactor replace object lookup
commit: don't lazy-fetch commits

 commit.c       | 18 ++++++++++++++--
 object-file.c  | 57 ++++++++++++++++++++++++++++++++++----------------
 object-store.h | 10 +++++++++
 3 files changed, 65 insertions(+), 20 deletions(-)
Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#1 lore raw

PATCH 1/4 object-file: reread object with exact same args

When an object in do_oid_object_info_extended() is found in a packfile, but corrupt, that packfile entry is marked as bad and the read is retried. Currently, this is done by invoking the function again but with the replace target of the object and with no flags.

This currently works, but will be clumsy when a later patch modifies this function to also return the "real" object being read (that is, the replace target). It does not make sense to pass a pointer in order to receive this information when no replace lookups are requested, which is exactly what the reinvocation does.

Therefore, change this reinvocation to pass exactly the arguments which were originally passed. This also makes us forwards compatible with future flags that may change the behavior of this function. This does slow down the case when packfile corruption is detected, but that is expected to be a very rare case.

---
 object-file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
object-file.c +1 -1
index 26290554bb..1cde477267 100644
@@ -1621,7 +1621,7 @@ static int do_oid_object_info_extended(struct repository *r,
 	rtype = packed_object_info(r, e.p, e.offset, oi);
 	if (rtype < 0) {
 		mark_bad_packed_object(e.p, real);
-		return do_oid_object_info_extended(r, real, oi, 0);
+		return do_oid_object_info_extended(r, oid, oi, flags);
 	} else if (oi->whence == OI_PACKED) {
 		oi->u.packed.offset = e.offset;
 		oi->u.packed.pack = e.p;
Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#2 lore raw

PATCH 2/4 object-file: refactor corrupt object diagnosis

In reply to Jonathan Tan

This functionality will be used from another file in a subsequent patch, so refactor it into a public function.

---
 object-file.c  | 29 ++++++++++++++++++-----------
 object-store.h |  9 +++++++++
 2 files changed, 27 insertions(+), 11 deletions(-)
object-file.c +18 -11
index 1cde477267..37468bc256 100644
@@ -1705,9 +1705,6 @@ void *read_object_file_extended(struct repository *r,
 				int lookup_replace)
 {
 	void *data;
-	const struct packed_git *p;
-	const char *path;
-	struct stat st;
 	const struct object_id *repl = lookup_replace ?
 		lookup_replace_object(r, oid) : oid;
 
@@ -1715,26 +1712,36 @@ void *read_object_file_extended(struct repository *r,
 	data = read_object(r, repl, type, size);
 	if (data)
 		return data;
+	die_if_corrupt(r, oid, repl);
+
+	return NULL;
+}
+
+void die_if_corrupt(struct repository *r,
+		    const struct object_id *oid,
+		    const struct object_id *real_oid)
+{
+	const struct packed_git *p;
+	const char *path;
+	struct stat st;
 
 	obj_read_lock();
 	if (errno && errno != ENOENT)
 		die_errno(_("failed to read object %s"), oid_to_hex(oid));
 
 	/* die if we replaced an object with one that does not exist */
-	if (repl != oid)
+	if (real_oid != oid)
 		die(_("replacement %s not found for %s"),
-		    oid_to_hex(repl), oid_to_hex(oid));
+		    oid_to_hex(real_oid), oid_to_hex(oid));
 
-	if (!stat_loose_object(r, repl, &st, &path))
+	if (!stat_loose_object(r, real_oid, &st, &path))
 		die(_("loose object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), path);
+		    oid_to_hex(real_oid), path);
 
-	if ((p = has_packed_and_bad(r, repl)))
+	if ((p = has_packed_and_bad(r, real_oid)))
 		die(_("packed object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), p->pack_name);
+		    oid_to_hex(real_oid), p->pack_name);
 	obj_read_unlock();
-
-	return NULL;
 }
 
 void *read_object_with_reference(struct repository *r,
object-store.h +9 -0
index 1be57abaf1..88c879c61e 100644
@@ -256,6 +256,15 @@ static inline void *repo_read_object_file(struct repository *r,
 #define read_object_file(oid, type, size) repo_read_object_file(the_repository, oid, type, size)
 #endif
 
+/*
+ * Dies if real_oid is corrupt, not just missing.
+ *
+ * real_oid should be an oid that could not be read.
+ */
+void die_if_corrupt(struct repository *r,
+		    const struct object_id *oid,
+		    const struct object_id *real_oid);
+
 /* Read and unpack an object file into memory, write memory to an object file */
 int oid_object_info(struct repository *r, const struct object_id *, unsigned long *);
 
Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#6 lore raw

PATCH 3/4 object-file: refactor replace object lookup

In reply to Jonathan Tan

Move the replace object lookup (specifically, the ability for the caller to know the result of the lookup) from read_object_file_extended() to one of the functions that it indirectly calls, do_oid_object_info_extended(), because a subsequent patch will need that ability from the latter.

---
 object-file.c  | 28 +++++++++++++++++++++-------
 object-store.h |  1 +
 2 files changed, 22 insertions(+), 7 deletions(-)
object-file.c +21 -7
index 37468bc256..fd394f1ace 100644
@@ -1546,6 +1546,11 @@ static int do_oid_object_info_extended(struct repository *r,
 
 	if (flags & OBJECT_INFO_LOOKUP_REPLACE)
 		real = lookup_replace_object(r, oid);
+	if (oi && oi->real_oidp) {
+		if (!(flags & OBJECT_INFO_LOOKUP_REPLACE))
+			BUG("specifying real_oidp does not make sense without OBJECT_INFO_LOOKUP_REPLACE");
+		*oi->real_oidp = real;
+	}
 
 	if (is_null_oid(real))
 		return -1;
@@ -1659,17 +1664,27 @@ int oid_object_info(struct repository *r,
 	return type;
 }
 
+/*
+ * If real_oid is not NULL, check if oid has a replace object and store the
+ * object that we end up using there.
+ */
 static void *read_object(struct repository *r,
 			 const struct object_id *oid, enum object_type *type,
-			 unsigned long *size)
+			 unsigned long *size, const struct object_id **real_oid)
 {
 	struct object_info oi = OBJECT_INFO_INIT;
 	void *content;
+	unsigned int flags = 0;
 	oi.typep = type;
 	oi.sizep = size;
 	oi.contentp = &content;
 
-	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
+	if (real_oid) {
+		flags |= OBJECT_INFO_LOOKUP_REPLACE;
+		oi.real_oidp = real_oid;
+	}
+
+	if (oid_object_info_extended(r, oid, &oi, flags) < 0)
 		return NULL;
 	return content;
 }
@@ -1705,14 +1720,13 @@ void *read_object_file_extended(struct repository *r,
 				int lookup_replace)
 {
 	void *data;
-	const struct object_id *repl = lookup_replace ?
-		lookup_replace_object(r, oid) : oid;
+	const struct object_id *real_oid;
 
 	errno = 0;
-	data = read_object(r, repl, type, size);
+	data = read_object(r, oid, type, size, &real_oid);
 	if (data)
 		return data;
-	die_if_corrupt(r, oid, repl);
+	die_if_corrupt(r, oid, real_oid);
 
 	return NULL;
 }
@@ -2283,7 +2297,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 
 	if (has_loose_object(oid))
 		return 0;
-	buf = read_object(the_repository, oid, &type, &len);
+	buf = read_object(the_repository, oid, &type, &len, NULL);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
 	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
object-store.h +1 -0
index 88c879c61e..9684562eb2 100644
@@ -406,6 +406,7 @@ struct object_info {
 	struct object_id *delta_base_oid;
 	struct strbuf *type_name;
 	void **contentp;
+	const struct object_id **real_oidp;
 
 	/* Response */
 	enum {
Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#8 lore raw

PATCH 4/4 commit: don't lazy-fetch commits

In reply to Jonathan Tan

When parsing commits, fail fast when the commit is missing or corrupt, instead of attempting to fetch them. This is done by inlining repo_read_object_file() and setting the flag that prevents fetching.

This is motivated by a situation in which through a bug (not necessarily through Git), there was corruption in the object store of a partial clone. In this particular case, the problem was exposed when "git gc" tried to expire reflogs, which calls repo_parse_commit(), which triggers fetches of the missing commits.

(There are other possible solutions to this problem including passing an argument from "git gc" to "git reflog" to inhibit all lazy fetches, but I think that this fix is at the wrong level - fixing "git reflog" means that this particular command works fine, or so we think (it will fail if it somehow needs to read a legitimately missing blob, say, a .gitmodules file), but fixing repo_parse_commit() will fix a whole class of bugs.)

---
 commit.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
commit.c +16 -2
index 572301b80a..17e71f5be4 100644
@@ -508,6 +508,13 @@ int repo_parse_commit_internal(struct repository *r,
 	enum object_type type;
 	void *buffer;
 	unsigned long size;
+	const struct object_id *real_oid;
+	struct object_info oi = {
+		.typep = &type,
+		.sizep = &size,
+		.contentp = &buffer,
+		.real_oidp = &real_oid,
+	};
 	int ret;
 
 	if (!item)
@@ -516,11 +523,18 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	/*
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	if (oid_object_info_extended(r, &item->object.oid, &oi,
+	    OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT) < 0) {
+		die_if_corrupt(r, &item->object.oid, real_oid);
 		return quiet_on_missing ? -1 :
 			error("Could not read %s",
 			     oid_to_hex(&item->object.oid));
+	}
 	if (type != OBJ_COMMIT) {
 		free(buffer);
 		return error("Object %s not a commit",
Signature · 1 line
2.38.1.584.g0f3c55d4c2-goog
#3 lore raw

Re: [PATCH 2/4] object-file: refactor corrupt object diagnosis

In reply to Jonathan Tan
Quoted 8 lines · Jonathan Tan
object-file.c +7 -0
+void die_if_corrupt(struct repository *r,
+		    const struct object_id *oid,
+		    const struct object_id *real_oid)
+{
+	const struct packed_git *p;
+	const char *path;
+	struct stat st;
 

object-file.c void *read_object_file_extended(struct repository *r, · line 1735

Quoted 10 lines
object-file.c +2 -2
 	obj_read_lock();
 	if (errno && errno != ENOENT)
 		die_errno(_("failed to read object %s"), oid_to_hex(oid));
 
 	/* die if we replaced an object with one that does not exist */
-	if (repl != oid)
+	if (real_oid != oid)
 		die(_("replacement %s not found for %s"),
-		    oid_to_hex(repl), oid_to_hex(oid));
+		    oid_to_hex(real_oid), oid_to_hex(oid));

This kind of pointer comparison is a little subtle. Within a single function, as this code was before this patch, it's probably OK to assume that we use pointer indirection, and a non-replaced object will use the original pointer. But for a public function, it seems like a gotcha that:

  oidcpy(&real_oid, lookup_replace_object(r, &oid));
  die_if_corrupt(r, &oid, &real_oid);

would produce the wrong answer (it would think replacement happened even if it didn't).

So maybe:

if (!oideq(real_oid, oid))

instead? It's a little slower, but the point of this is to diagnose and die, so it's not exactly a hot path. :)

-Peff

#7 lore raw

Re: [PATCH 3/4] object-file: refactor replace object lookup

In reply to Jonathan Tan

object-store.h struct object_info { · line 409

Quoted 9 lines · Jonathan Tan
object-store.h +1 -0
index 88c879c61e..9684562eb2 100644
@@ -406,6 +406,7 @@ struct object_info {
 	struct object_id *delta_base_oid;
 	struct strbuf *type_name;
 	void **contentp;
+	const struct object_id **real_oidp;

OK. The double-pointer here is a bit funky as an interface. It may point back to the "oid" we fed the function, or it may point to long-term storage owned by the replace mechanism. A more straightforward one would be to store a single-pointer to caller-owned storage, and copy to that (like we do for delta_base_oid, for example).

But doing it this way avoids extra oid copies in the normal, non-replaced case, and matches how the current callers view things. So while it's a little convoluted, I think it makes sense to do it as you did.

-Peff

#9 lore raw

Re: [PATCH 4/4] commit: don't lazy-fetch commits

In reply to Jonathan Tan
Quoted 9 lines · Jonathan Tan

When parsing commits, fail fast when the commit is missing or corrupt, instead of attempting to fetch them. This is done by inlining repo_read_object_file() and setting the flag that prevents fetching.

This is motivated by a situation in which through a bug (not necessarily through Git), there was corruption in the object store of a partial clone. In this particular case, the problem was exposed when "git gc" tried to expire reflogs, which calls repo_parse_commit(), which triggers fetches of the missing commits.

Makes sense.

Quoted 26 lines
commit.c +9 -2
index 572301b80a..17e71f5be4 100644
@@ -508,6 +508,13 @@ int repo_parse_commit_internal(struct repository *r,
 	enum object_type type;
 	void *buffer;
 	unsigned long size;
+	const struct object_id *real_oid;
+	struct object_info oi = {
+		.typep = &type,
+		.sizep = &size,
+		.contentp = &buffer,
+		.real_oidp = &real_oid,
+	};
 	int ret;
 
 	if (!item)
@@ -516,11 +523,18 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	/*

commit.c int repo_parse_commit_internal(struct repository *r, · line 537

Quoted 10 lines
commit.c +7 -0
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	if (oid_object_info_extended(r, &item->object.oid, &oi,
+	    OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT) < 0) {
+		die_if_corrupt(r, &item->object.oid, real_oid);
 		return quiet_on_missing ? -1 :
 			error("Could not read %s",
 			     oid_to_hex(&item->object.oid));
+	}

OK, so we know we want a commit object because we're in the commit-parsing function, so we just ask to disable fetching.

Two devil's advocate thoughts:

1. What if we're wrong that it's a commit? If somebody references a blob in a commit "parent" header, the "right" outcome is for us to say "oops, the type is wrong" when we try to parse it as a commit. But now in a partial clone, we might avoid fetching that supposed commit and say "you don't have this object", even though we could get it.

I think I'm OK with that. Either way, the repo is corrupt, and we'll have informed the user of that. The fact that this bizarre and specific sequence of corruptions might not go as far as it can to deduce the root cause is probably fine.

2. Are there other places where we'd want to do the same thing? E.g., in parse_object() we might ask for an object (not knowing its type) only to find out that it is a commit. But we have no idea if we lazy-fetched it or not!

I had somehow imagined your series would be hooking in at the level of the lazy-fetch code, and complaining about fetching commits. But that may be tricky to do, because we really don't know the type until after we fetch it, and selectively removing an object from the odb is quite hard. We'd probably have to tell the other side "please, don't send me any commits", which requires a protocol extension, and...yuck.

By comparison, your approach is an easy win that may catch problems in practice (and is certainly better than the status quo).

-Peff

#13 lore raw
In reply to Jonathan Tan
Quoted 8 lines · Jonathan Tan

This is a follow-up from my previous email about the possibility of not fetching when we know that we're fetching a commit [1]. I had to refactor a few things mostly due to replace objects, so the number of patches might be larger than you would expect. I tried to keep each patch small and easy to understand, though.

Patches 1-3 contain some forward-compatibility and refactoring changes, and patch 4 contains the actual logic change.

These look pretty good to me. I raised a minor nit in patch 2; if you agree it should be a trivial re-roll.

I left some thoughts on the approach in patch 4, but I think given that this is a strict improvement over the status quo, it's a good step forward, even if it won't catch all such cases.

-Peff

#4 lore raw

Re: [PATCH 2/4] object-file: refactor corrupt object diagnosis

In reply to Jeff King
Quoted 6 lines · Jeff King

So maybe:

if (!oideq(real_oid, oid))

instead? It's a little slower, but the point of this is to diagnose and die, so it's not exactly a hot path. :)

Very true, including the part that the original is fine because it is localized and fairly obvious. For a public function, we cannot assume any additional constraints between oid and real_oid (other than they are of the same "struct object_id *" type) like the two pointers prepared locally in the original had, and use of oideq() would be more appropriate here.

Thanks.

#12 lore raw

Re: [PATCH 4/4] commit: don't lazy-fetch commits

In reply to Jonathan Tan
Quoted 11 lines · Jonathan Tan

When parsing commits, fail fast when the commit is missing or corrupt, instead of attempting to fetch them. This is done by inlining repo_read_object_file() and setting the flag that prevents fetching.

This is motivated by a situation in which through a bug (not necessarily through Git), there was corruption in the object store of a partial clone. In this particular case, the problem was exposed when "git gc" tried to expire reflogs, which calls repo_parse_commit(), which triggers fetches of the missing commits.

The assumption is that there will never be a filtering mode that says "give us tags and we will lazy-fetch everything reachable from it when we need it", with which the "solution" will break down, I think, and I would say it is probably good enough.

#5 lore raw

Re: [PATCH 2/4] object-file: refactor corrupt object diagnosis

In reply to Junio C Hamano
Quoted 9 lines · Junio C Hamano
Quoted 6 lines · Jeff King

So maybe:

if (!oideq(real_oid, oid))

instead? It's a little slower, but the point of this is to diagnose and die, so it's not exactly a hot path. :)

Quoted 8 lines

Very true, including the part that the original is fine because it is localized and fairly obvious. For a public function, we cannot assume any additional constraints between oid and real_oid (other than they are of the same "struct object_id *" type) like the two pointers prepared locally in the original had, and use of oideq() would be more appropriate here.

Thanks.

Makes sense. I'll make the change.

#10 lore raw

Re: [PATCH 4/4] commit: don't lazy-fetch commits

In reply to Jeff King
Quoted 30 lines · Jeff King

OK, so we know we want a commit object because we're in the commit-parsing function, so we just ask to disable fetching.

Two devil's advocate thoughts:

1. What if we're wrong that it's a commit? If somebody references a blob in a commit "parent" header, the "right" outcome is for us to say "oops, the type is wrong" when we try to parse it as a commit. But now in a partial clone, we might avoid fetching that supposed commit and say "you don't have this object", even though we could get it.

I think I'm OK with that. Either way, the repo is corrupt, and we'll have informed the user of that. The fact that this bizarre and specific sequence of corruptions might not go as far as it can to deduce the root cause is probably fine.

2. Are there other places where we'd want to do the same thing? E.g., in parse_object() we might ask for an object (not knowing its type) only to find out that it is a commit. But we have no idea if we lazy-fetched it or not!

I had somehow imagined your series would be hooking in at the level of the lazy-fetch code, and complaining about fetching commits. But that may be tricky to do, because we really don't know the type until after we fetch it, and selectively removing an object from the odb is quite hard. We'd probably have to tell the other side "please, don't send me any commits", which requires a protocol extension, and...yuck.

Quoted 4 lines

By comparison, your approach is an easy win that may catch problems in practice (and is certainly better than the status quo).

-Peff

Thanks for taking a look. Let me know if you think that the commit message could be improved to cover these cases. Right now I think that e.g. "When parsing an object believed to be a commit in repo_parse_commit_internal()" instead of "When parsing commits" wouldn't add much value, but I might be missing something.

#14 lore raw

v2 cover Don't lazy-fetch commits when parsing them

In reply to Jonathan Tan

Thanks everyone for your reviews. Here is a reroll with the requested change (just one small one).

Jonathan Tan (4):
object-file: reread object with exact same args
object-file: refactor corrupt object diagnosis
object-file: refactor replace object lookup
commit: don't lazy-fetch commits

 commit.c       | 18 ++++++++++++++--
 object-file.c  | 57 ++++++++++++++++++++++++++++++++++----------------
 object-store.h | 10 +++++++++
 3 files changed, 65 insertions(+), 20 deletions(-)

Range-diff against v1:

1:  604160e79c = 1:  604160e79c object-file: reread object with exact same args
2:  a8c5fcd9f8 ! 2:  1be60f1bf2 object-file: refactor corrupt object diagnosis
    @@ object-file.c: void *read_object_file_extended(struct repository *r,
      
      	/* die if we replaced an object with one that does not exist */
     -	if (repl != oid)
    -+	if (real_oid != oid)
    ++	if (!oideq(real_oid, oid))
      		die(_("replacement %s not found for %s"),
     -		    oid_to_hex(repl), oid_to_hex(oid));
     +		    oid_to_hex(real_oid), oid_to_hex(oid));
3:  940396307f = 3:  28935ba1b0 object-file: refactor replace object lookup
4:  6af8dcebd1 = 4:  a38229c42a commit: don't lazy-fetch commits
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
#15 lore raw

PATCH v2 1/4 object-file: reread object with exact same args

When an object in do_oid_object_info_extended() is found in a packfile, but corrupt, that packfile entry is marked as bad and the read is retried. Currently, this is done by invoking the function again but with the replace target of the object and with no flags.

This currently works, but will be clumsy when a later patch modifies this function to also return the "real" object being read (that is, the replace target). It does not make sense to pass a pointer in order to receive this information when no replace lookups are requested, which is exactly what the reinvocation does.

Therefore, change this reinvocation to pass exactly the arguments which were originally passed. This also makes us forwards compatible with future flags that may change the behavior of this function. This does slow down the case when packfile corruption is detected, but that is expected to be a very rare case.

---
 object-file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
object-file.c +1 -1
index 26290554bb..1cde477267 100644
@@ -1621,7 +1621,7 @@ static int do_oid_object_info_extended(struct repository *r,
 	rtype = packed_object_info(r, e.p, e.offset, oi);
 	if (rtype < 0) {
 		mark_bad_packed_object(e.p, real);
-		return do_oid_object_info_extended(r, real, oi, 0);
+		return do_oid_object_info_extended(r, oid, oi, flags);
 	} else if (oi->whence == OI_PACKED) {
 		oi->u.packed.offset = e.offset;
 		oi->u.packed.pack = e.p;
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
#17 lore raw

PATCH v2 3/4 object-file: refactor replace object lookup

In reply to Jonathan Tan

Move the replace object lookup (specifically, the ability for the caller to know the result of the lookup) from read_object_file_extended() to one of the functions that it indirectly calls, do_oid_object_info_extended(), because a subsequent patch will need that ability from the latter.

---
 object-file.c  | 28 +++++++++++++++++++++-------
 object-store.h |  1 +
 2 files changed, 22 insertions(+), 7 deletions(-)
object-file.c +21 -7
index 36f81c7958..8adef99a7c 100644
@@ -1546,6 +1546,11 @@ static int do_oid_object_info_extended(struct repository *r,
 
 	if (flags & OBJECT_INFO_LOOKUP_REPLACE)
 		real = lookup_replace_object(r, oid);
+	if (oi && oi->real_oidp) {
+		if (!(flags & OBJECT_INFO_LOOKUP_REPLACE))
+			BUG("specifying real_oidp does not make sense without OBJECT_INFO_LOOKUP_REPLACE");
+		*oi->real_oidp = real;
+	}
 
 	if (is_null_oid(real))
 		return -1;
@@ -1659,17 +1664,27 @@ int oid_object_info(struct repository *r,
 	return type;
 }
 
+/*
+ * If real_oid is not NULL, check if oid has a replace object and store the
+ * object that we end up using there.
+ */
 static void *read_object(struct repository *r,
 			 const struct object_id *oid, enum object_type *type,
-			 unsigned long *size)
+			 unsigned long *size, const struct object_id **real_oid)
 {
 	struct object_info oi = OBJECT_INFO_INIT;
 	void *content;
+	unsigned int flags = 0;
 	oi.typep = type;
 	oi.sizep = size;
 	oi.contentp = &content;
 
-	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
+	if (real_oid) {
+		flags |= OBJECT_INFO_LOOKUP_REPLACE;
+		oi.real_oidp = real_oid;
+	}
+
+	if (oid_object_info_extended(r, oid, &oi, flags) < 0)
 		return NULL;
 	return content;
 }
@@ -1705,14 +1720,13 @@ void *read_object_file_extended(struct repository *r,
 				int lookup_replace)
 {
 	void *data;
-	const struct object_id *repl = lookup_replace ?
-		lookup_replace_object(r, oid) : oid;
+	const struct object_id *real_oid;
 
 	errno = 0;
-	data = read_object(r, repl, type, size);
+	data = read_object(r, oid, type, size, &real_oid);
 	if (data)
 		return data;
-	die_if_corrupt(r, oid, repl);
+	die_if_corrupt(r, oid, real_oid);
 
 	return NULL;
 }
@@ -2283,7 +2297,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 
 	if (has_loose_object(oid))
 		return 0;
-	buf = read_object(the_repository, oid, &type, &len);
+	buf = read_object(the_repository, oid, &type, &len, NULL);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
 	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
object-store.h +1 -0
index 88c879c61e..9684562eb2 100644
@@ -406,6 +406,7 @@ struct object_info {
 	struct object_id *delta_base_oid;
 	struct strbuf *type_name;
 	void **contentp;
+	const struct object_id **real_oidp;
 
 	/* Response */
 	enum {
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
#16 lore raw

PATCH v2 2/4 object-file: refactor corrupt object diagnosis

In reply to Jonathan Tan

This functionality will be used from another file in a subsequent patch, so refactor it into a public function.

---
 object-file.c  | 29 ++++++++++++++++++-----------
 object-store.h |  9 +++++++++
 2 files changed, 27 insertions(+), 11 deletions(-)
object-file.c +18 -11
index 1cde477267..36f81c7958 100644
@@ -1705,9 +1705,6 @@ void *read_object_file_extended(struct repository *r,
 				int lookup_replace)
 {
 	void *data;
-	const struct packed_git *p;
-	const char *path;
-	struct stat st;
 	const struct object_id *repl = lookup_replace ?
 		lookup_replace_object(r, oid) : oid;
 
@@ -1715,26 +1712,36 @@ void *read_object_file_extended(struct repository *r,
 	data = read_object(r, repl, type, size);
 	if (data)
 		return data;
+	die_if_corrupt(r, oid, repl);
+
+	return NULL;
+}
+
+void die_if_corrupt(struct repository *r,
+		    const struct object_id *oid,
+		    const struct object_id *real_oid)
+{
+	const struct packed_git *p;
+	const char *path;
+	struct stat st;
 
 	obj_read_lock();
 	if (errno && errno != ENOENT)
 		die_errno(_("failed to read object %s"), oid_to_hex(oid));
 
 	/* die if we replaced an object with one that does not exist */
-	if (repl != oid)
+	if (!oideq(real_oid, oid))
 		die(_("replacement %s not found for %s"),
-		    oid_to_hex(repl), oid_to_hex(oid));
+		    oid_to_hex(real_oid), oid_to_hex(oid));
 
-	if (!stat_loose_object(r, repl, &st, &path))
+	if (!stat_loose_object(r, real_oid, &st, &path))
 		die(_("loose object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), path);
+		    oid_to_hex(real_oid), path);
 
-	if ((p = has_packed_and_bad(r, repl)))
+	if ((p = has_packed_and_bad(r, real_oid)))
 		die(_("packed object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), p->pack_name);
+		    oid_to_hex(real_oid), p->pack_name);
 	obj_read_unlock();
-
-	return NULL;
 }
 
 void *read_object_with_reference(struct repository *r,
object-store.h +9 -0
index 1be57abaf1..88c879c61e 100644
@@ -256,6 +256,15 @@ static inline void *repo_read_object_file(struct repository *r,
 #define read_object_file(oid, type, size) repo_read_object_file(the_repository, oid, type, size)
 #endif
 
+/*
+ * Dies if real_oid is corrupt, not just missing.
+ *
+ * real_oid should be an oid that could not be read.
+ */
+void die_if_corrupt(struct repository *r,
+		    const struct object_id *oid,
+		    const struct object_id *real_oid);
+
 /* Read and unpack an object file into memory, write memory to an object file */
 int oid_object_info(struct repository *r, const struct object_id *, unsigned long *);
 
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
#18 lore raw

PATCH v2 4/4 commit: don't lazy-fetch commits

In reply to Jonathan Tan

When parsing commits, fail fast when the commit is missing or corrupt, instead of attempting to fetch them. This is done by inlining repo_read_object_file() and setting the flag that prevents fetching.

This is motivated by a situation in which through a bug (not necessarily through Git), there was corruption in the object store of a partial clone. In this particular case, the problem was exposed when "git gc" tried to expire reflogs, which calls repo_parse_commit(), which triggers fetches of the missing commits.

(There are other possible solutions to this problem including passing an argument from "git gc" to "git reflog" to inhibit all lazy fetches, but I think that this fix is at the wrong level - fixing "git reflog" means that this particular command works fine, or so we think (it will fail if it somehow needs to read a legitimately missing blob, say, a .gitmodules file), but fixing repo_parse_commit() will fix a whole class of bugs.)

---
 commit.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)
commit.c +16 -2
index 572301b80a..17e71f5be4 100644
@@ -508,6 +508,13 @@ int repo_parse_commit_internal(struct repository *r,
 	enum object_type type;
 	void *buffer;
 	unsigned long size;
+	const struct object_id *real_oid;
+	struct object_info oi = {
+		.typep = &type,
+		.sizep = &size,
+		.contentp = &buffer,
+		.real_oidp = &real_oid,
+	};
 	int ret;
 
 	if (!item)
@@ -516,11 +523,18 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	/*
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	if (oid_object_info_extended(r, &item->object.oid, &oi,
+	    OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT) < 0) {
+		die_if_corrupt(r, &item->object.oid, real_oid);
 		return quiet_on_missing ? -1 :
 			error("Could not read %s",
 			     oid_to_hex(&item->object.oid));
+	}
 	if (type != OBJ_COMMIT) {
 		free(buffer);
 		return error("Object %s not a commit",
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
#11 lore raw

Re: [PATCH 4/4] commit: don't lazy-fetch commits

In reply to Jonathan Tan
Quoted 4 lines · Jonathan Tan
Quoted 3 lines · Jeff King

OK, so we know we want a commit object because we're in the commit-parsing function, so we just ask to disable fetching.

Quoted 8 lines
Quoted 1 line

Two devil's advocate thoughts:

[...]

Thanks for taking a look. Let me know if you think that the commit message could be improved to cover these cases. Right now I think that e.g. "When parsing an object believed to be a commit in repo_parse_commit_internal()" instead of "When parsing commits" wouldn't add much value, but I might be missing something.

I think your commit message is OK as-is. I was mostly just laying out my thoughts in reviewing. Some of that could go into the commit message as notes, but I think it is sufficient that they're here in the list archive.

-Peff

#19 lore raw
In reply to Jonathan Tan
Quoted 2 lines · Jonathan Tan

Thanks everyone for your reviews. Here is a reroll with the requested change (just one small one).

Thanks, this looks OK to me. However Junio noted in "What's cooking" that it seems to break CI on windows. The problem is in t5318.93:

  2022-12-01T09:26:44.8887018Z ++ cat test_err
  2022-12-01T09:26:44.8887414Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8887825Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8888240Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8888639Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8889052Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8889512Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8889991Z fatal: failed to read object 0000000000000000000000000000000000000000: Function not implemented
  2022-12-01T09:26:44.8890401Z ++ return 1
  2022-12-01T09:26:44.8890761Z error: last command exited with $?=1
  2022-12-01T09:26:44.8891263Z not ok 93 - corrupt commit-graph write (broken parent)

Looks like the check in die_if_corrupt() is seeing a different errno value than ENOENT. I wonder if we need to take more care to preserve it across calls. It does look like we hit the same sequence of functions that read_object_file_extended() did, but perhaps this was buggy all along, and you're now exposing it through a new code path.

In particular I wonder if obj_read_unlock() might be the culprit here, and something like this might help:

object-file.c +3 -0
index 8adef99a7c..db2d35519e 100644
@@ -1641,9 +1641,12 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid,
 			     struct object_info *oi, unsigned flags)
 {
 	int ret;
+	int save_errno;
 	obj_read_lock();
 	ret = do_oid_object_info_extended(r, oid, oi, flags);
+	save_errno = errno;
 	obj_read_unlock();
+	errno = save_errno;
 	return ret;
 }
 

-Peff

#20 lore raw
Quoted 35 lines · Jeff King
Quoted 2 lines · Jonathan Tan

Thanks everyone for your reviews. Here is a reroll with the requested change (just one small one).

Thanks, this looks OK to me. However Junio noted in "What's cooking" that it seems to break CI on windows. The problem is in t5318.93:

  2022-12-01T09:26:44.8887018Z ++ cat test_err
  2022-12-01T09:26:44.8887414Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8887825Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8888240Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8888639Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8889052Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8889512Z error: Could not read 0000000000000000000000000000000000000000
  2022-12-01T09:26:44.8889991Z fatal: failed to read object 0000000000000000000000000000000000000000: Function not implemented
  2022-12-01T09:26:44.8890401Z ++ return 1
  2022-12-01T09:26:44.8890761Z error: last command exited with $?=1
  2022-12-01T09:26:44.8891263Z not ok 93 - corrupt commit-graph write (broken parent)

Looks like the check in die_if_corrupt() is seeing a different errno value than ENOENT. I wonder if we need to take more care to preserve it across calls. It does look like we hit the same sequence of functions that read_object_file_extended() did, but perhaps this was buggy all along, and you're now exposing it through a new code path.

In particular I wonder if obj_read_unlock() might be the culprit here, and something like this might help:

object-file.c +0 -0
index 8adef99a7c..db2d35519e 100644
@@ -1641,9 +1641,12 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid,
 			     struct object_info *oi, unsigned flags)

object-file.c int oid_object_info_extended(struct repository *r, const struct object_id *oid, · line 1651

Quoted 10 lines
object-file.c +3 -0
 {
 	int ret;
+	int save_errno;
 	obj_read_lock();
 	ret = do_oid_object_info_extended(r, oid, oi, flags);
+	save_errno = errno;
 	obj_read_unlock();
+	errno = save_errno;
 	return ret;
 }

Copying die_if_corrupt() until "failed to read object":

Quoted 11 lines
object-file.c +0 -0
 1734 void die_if_corrupt(struct repository *r,                                                                                                                                                       

1735 const struct object_id *oid,
1736 const struct object_id *real_oid)
1737 {

1738         const struct packed_git *p;                                                                                                                                                             
1739         const char *path;                                                                                                                                                                       
1740         struct stat st;                                                                                                                                                                         

1741
1742 obj_read_lock();
1743 if (errno && errno != ENOENT)
1744 die_errno(_("failed to read object %s"), oid_to_hex(oid));

I wonder if we could just remove this check. Even as it is, I don't think that there is any guarantee that obj_read_lock() would not clobber errno. Removing it makes all tests pass locally, but I haven't tried it on CI.

(One argument that could be made is that we shouldn't have any die_if_corrupt() refactoring or other refactoring of the sort, because previously its contents was part of a function and it could thus rely on the errno of what has happened previously. But I think that even without my patches, we couldn't rely on it in the first place - looking at obj_read_lock(), it looks like it could init a mutex, and depending on the implementation of that, it could clobber errno.)

#24 lore raw
In reply to Jeff King
Quoted 10 lines · Jeff King
Quoted 2 lines · Jonathan Tan

Thanks everyone for your reviews. Here is a reroll with the requested change (just one small one).

Thanks, this looks OK to me. However Junio noted in "What's cooking" that it seems to break CI on windows. The problem is in t5318.93:
...
In particular I wonder if obj_read_unlock() might be the culprit here, and something like this might help:

Thanks for following-up.

#21 lore raw
In reply to Jonathan Tan
Quoted 16 lines · Jonathan Tan
Quoted 11 lines

1734 void die_if_corrupt(struct repository *r,
1735 const struct object_id *oid,
1736 const struct object_id *real_oid)
1737 {

1738         const struct packed_git *p;
1739         const char *path;
1740         struct stat st;

1741
1742 obj_read_lock();
1743 if (errno && errno != ENOENT)
1744 die_errno(_("failed to read object %s"), oid_to_hex(oid));

I wonder if we could just remove this check. Even as it is, I don't think that there is any guarantee that obj_read_lock() would not clobber errno. Removing it makes all tests pass locally, but I haven't tried it on CI.

Quoted 6 lines

(One argument that could be made is that we shouldn't have any die_if_corrupt() refactoring or other refactoring of the sort, because previously its contents was part of a function and it could thus rely on the errno of what has happened previously. But I think that even without my patches, we couldn't rely on it in the first place - looking at obj_read_lock(), it looks like it could init a mutex, and depending on the implementation of that, it could clobber errno.)

Yeah, I don't see any difference in the new caller versus what the original was doing. The errno we care about comes from inside oid_object_info_extended(). So in any case, we'll see at least obj_read_unlock() followed by obj_read_lock() between the syscalls of interest and this check. And I don't even really see any indication that oid_object_info_extended() tries to set or preserve errno itself. The likely sequence is:

- find_pack_entry() fails to find it; errno isn't set at all
- loose_object_info() tries to open it and probably gets ENOENT
- we check find_pack_entry() again after reprepare_packed_git()
- that fails so we return -1, barring submodule or partial clone tricks

So it really seems like we're quite likely to get an errno from opening or mapping packs. Which implies the original suffers from the same issue, but we simply never triggered it meaningfully in a test.

I'm not entirely sure on just removing the check. It comes from 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28), so we'd lose what that commit is trying to do. Though I think even back then, I think it would have suffered from the same problems (minus the lock/unlock; I'm still unclear which syscall is the actual culprit here).

If we assume that errno from reading the object isn't reliable, I think you'd have to actually re-check things. Something like:

  if (find_pack_entry(...) || !stat_loose_object(...))
    /* ok, it's not missing */

but of course we don't have the actual errno that _did_ cause us to fail, which makes the error message we'd print a lot less useful. Maybe this check should be ditched and we should complain much closer to the source of the problem:

object-file.c +5 -1
index 26290554bb..743ba8210e 100644
@@ -1460,8 +1460,12 @@ static int loose_object_info(struct repository *r,
 	}
 
 	map = map_loose_object(r, oid, &mapsize);
-	if (!map)
+	if (!map) {
+		if (errno != ENOENT)
+			error_errno("unable to open loose object %s",
+				    oid_to_hex(oid));
 		return -1;
+	}
 
 	if (!oi->sizep)
 		oi->sizep = &size_scratch;

That might make things more verbose for other code paths, but that kind of seems like a good thing. If you have an object file that we can't open, we probably _should_ be complaining loudly about it.

We may need to be a little more careful about preserving errno in map_loose_object_1(), though (gee, another place where the existing check could run into trouble).

-Peff

#22 lore raw
Quoted 14 lines · Jeff King

Yeah, I don't see any difference in the new caller versus what the original was doing. The errno we care about comes from inside oid_object_info_extended(). So in any case, we'll see at least obj_read_unlock() followed by obj_read_lock() between the syscalls of interest and this check. And I don't even really see any indication that oid_object_info_extended() tries to set or preserve errno itself. The likely sequence is:

- find_pack_entry() fails to find it; errno isn't set at all
- loose_object_info() tries to open it and probably gets ENOENT
- we check find_pack_entry() again after reprepare_packed_git()
- that fails so we return -1, barring submodule or partial clone tricks

Quoted 3 lines

So it really seems like we're quite likely to get an errno from opening or mapping packs. Which implies the original suffers from the same issue, but we simply never triggered it meaningfully in a test.

Thanks for checking. I'm still not sure how the current code passes CI, but my patches don't.

Quoted 6 lines

I'm not entirely sure on just removing the check. It comes from 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28), so we'd lose what that commit is trying to do. Though I think even back then, I think it would have suffered from the same problems (minus the lock/unlock; I'm still unclear which syscall is the actual culprit here).

Ah, thanks for the pointer to that commit. Without that, my patch would report corruption even if the real issue was EMFILE, as the commit message of that commit describes.

Quoted 30 lines

If we assume that errno from reading the object isn't reliable, I think you'd have to actually re-check things. Something like:

  if (find_pack_entry(...) || !stat_loose_object(...))
    /* ok, it's not missing */

but of course we don't have the actual errno that _did_ cause us to fail, which makes the error message we'd print a lot less useful. Maybe this check should be ditched and we should complain much closer to the source of the problem:

object-file.c +5 -1
index 26290554bb..743ba8210e 100644
@@ -1460,8 +1460,12 @@ static int loose_object_info(struct repository *r,
 	}
 
 	map = map_loose_object(r, oid, &mapsize);
-	if (!map)
+	if (!map) {
+		if (errno != ENOENT)
+			error_errno("unable to open loose object %s",
+				    oid_to_hex(oid));
 		return -1;
+	}
 
 	if (!oi->sizep)
 		oi->sizep = &size_scratch;
Quoted 7 lines

That might make things more verbose for other code paths, but that kind of seems like a good thing. If you have an object file that we can't open, we probably _should_ be complaining loudly about it.

We may need to be a little more careful about preserving errno in map_loose_object_1(), though (gee, another place where the existing check could run into trouble).

Besides needing to be careful in map_loose_object_1(), I'm not sure if this fully solves the problem. This is non-fatal, so the EMFILE commit's work would still remain undone. If this were made fatal, I think this would change the behavior of too much code, especially those that can tolerate loose objects being missing.

What do you think of not putting any die_if_corrupt() calls in the commit parsing code at all? The error message printed would then be different (just a generic message about being unable to parse a commit, versus the specific one here) but it does pass CI [1]. Also, I don't think that we should be doing errno diagnostics separate from what causes the errno anyway.

[1] https://github.com/jonathantanmy/git/actions/runs/3624495729

#23 lore raw
Quoted 6 lines · Jonathan Tan
Quoted 3 lines

So it really seems like we're quite likely to get an errno from opening or mapping packs. Which implies the original suffers from the same issue, but we simply never triggered it meaningfully in a test.

Thanks for checking. I'm still not sure how the current code passes CI, but my patches don't.

Hmm. Actually, I am now, too.

My assumption was that only certain tests would notice the problem, because both outcomes are an error, and they only differ in what stderr says ("this does not exist" versus "we got this weird errno"). And so I assumed that the old spot in read_object_file() did not happen to trigger any tests which check stderr, but your new caller in repo_parse_commit_internal() was unlucky enough to do so. But since that new caller was calling repo_read_object_file() before, I'd think it would have triggered the same thing.

Quoted 8 lines
Quoted 7 lines

That might make things more verbose for other code paths, but that kind of seems like a good thing. If you have an object file that we can't open, we probably _should_ be complaining loudly about it.

We may need to be a little more careful about preserving errno in map_loose_object_1(), though (gee, another place where the existing check could run into trouble).

Quoted 5 lines

Besides needing to be careful in map_loose_object_1(), I'm not sure if this fully solves the problem. This is non-fatal, so the EMFILE commit's work would still remain undone. If this were made fatal, I think this would change the behavior of too much code, especially those that can tolerate loose objects being missing.

True, in the sense that we'd still say "X is corrupt". But I think the real sin prior to 3ba7a0655 is that we _only_ said "hey, this looks corrupt". If the output is:

error: unable to mmap .git/objects/12/3456abcd...
fatal: object 123456abcd is corrupt or missing

then that is at least not actively misleading (and is broadly similar to other cases in Git, where higher-level code only knows "I expected us to have this object and for some reason we don't", but without knowing whether it was missing, or a system error, or corrupt).

That said I think all of these die() statements that you moved into die_if_corrupt() are already doing the wrong thing. We should probably mention errors (besides "missing object") to the user at the lowest level where we can give the most detail, and then return errors up the stack. That makes Git more verbose, but remember we're talking about corrupted or broken repositories here. You shouldn't see these under normal circumstances.

Quoted 5 lines

What do you think of not putting any die_if_corrupt() calls in the commit parsing code at all? The error message printed would then be different (just a generic message about being unable to parse a commit, versus the specific one here) but it does pass CI [1]. Also, I don't think that we should be doing errno diagnostics separate from what causes the errno anyway.

So I think that's a lesser version of what I'm proposing above. ;) In the sense that yes, I would say that repo_parse_commit_internal() does not need to do anything more specific than its existing "could not read" message. But I would go further and say:

- it would be nice the low-level code where errno _is_ valid said more about what happened

- we should take read_object_file_extended() in the same direction

-Peff

#25 lore raw

v2 cover Don't lazy-fetch commits when parsing them

In reply to Jonathan Tan

Thanks everyone for taking a look. In the end, I thought it best to bite the bullet and move all the corruption diagnostics to where they are detected instead of re-checking (and thus relying on errno to be preserved) after we have found out that we couldn't read the object. This does mean a reworking of all the earlier patches, but overall I think that this puts the code in a better state.

I have also verified that this passes CI [1].

[1] https://github.com/jonathantanmy/git/actions/runs/3634359088

Jonathan Tan (3):
object-file: don't exit early if skipping loose
object-file: emit corruption errors when detected
commit: don't lazy-fetch commits

 commit.c       | 15 +++++++--
 object-file.c  | 84 ++++++++++++++++++++++++++------------------------
 object-store.h |  3 ++
 3 files changed, 59 insertions(+), 43 deletions(-)

Range-diff against v1:

1:  3a00bc45fd < -:  ---------- object-file: reread object with exact same args
2:  9999e127a0 < -:  ---------- object-file: refactor corrupt object diagnosis
3:  28c7ee2f8c < -:  ---------- object-file: refactor replace object lookup
-:  ---------- > 1:  9ad34a1dce object-file: don't exit early if skipping loose
-:  ---------- > 2:  9ddfff3585 object-file: emit corruption errors when detected
4:  0607fa67d1 ! 3:  c5fe42deb0 commit: don't lazy-fetch commits
    @@ commit.c: int repo_parse_commit_internal(struct repository *r,
      	enum object_type type;
      	void *buffer;
      	unsigned long size;
    -+	const struct object_id *real_oid;
     +	struct object_info oi = {
     +		.typep = &type,
     +		.sizep = &size,
     +		.contentp = &buffer,
    -+		.real_oidp = &real_oid,
     +	};
    ++	/*
    ++	 * Git does not support partial clones that exclude commits, so set
    ++	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
    ++	 */
    ++	int flags = OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT |
    ++		OBJECT_INFO_DIE_IF_CORRUPT;
      	int ret;
      
      	if (!item)
    @@ commit.c: int repo_parse_commit_internal(struct repository *r,
     -	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
     -	if (!buffer)
     +
    -+	/*
    -+	 * Git does not support partial clones that exclude commits, so set
    -+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
    -+	 */
    -+	if (oid_object_info_extended(r, &item->object.oid, &oi,
    -+	    OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT) < 0) {
    -+		die_if_corrupt(r, &item->object.oid, real_oid);
    ++	if (oid_object_info_extended(r, &item->object.oid, &oi, flags) < 0)
      		return quiet_on_missing ? -1 :
      			error("Could not read %s",
      			     oid_to_hex(&item->object.oid));
    -+	}
    - 	if (type != OBJ_COMMIT) {
    - 		free(buffer);
    - 		return error("Object %s not a commit",
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
#26 lore raw

PATCH v2 1/3 object-file: don't exit early if skipping loose

Instead, also search the submodule object stores and promisor remotes.

This also centralizes what happens when the object is not found (the "return -1"), which is useful for a subsequent patch.

---
 object-file.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
object-file.c +11 -12
index 26290554bb..596dd049fd 100644
@@ -1575,18 +1575,17 @@ static int do_oid_object_info_extended(struct repository *r,
 		if (find_pack_entry(r, real, &e))
 			break;
 
-		if (flags & OBJECT_INFO_IGNORE_LOOSE)
-			return -1;
-
-		/* Most likely it's a loose object. */
-		if (!loose_object_info(r, real, oi, flags))
-			return 0;
-
-		/* Not a loose object; someone else may have just packed it. */
-		if (!(flags & OBJECT_INFO_QUICK)) {
-			reprepare_packed_git(r);
-			if (find_pack_entry(r, real, &e))
-				break;
+		if (!(flags & OBJECT_INFO_IGNORE_LOOSE)) {
+			/* Most likely it's a loose object. */
+			if (!loose_object_info(r, real, oi, flags))
+				return 0;
+
+			/* Not a loose object; someone else may have just packed it. */
+			if (!(flags & OBJECT_INFO_QUICK)) {
+				reprepare_packed_git(r);
+				if (find_pack_entry(r, real, &e))
+					break;
+			}
 		}
 
 		/*
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
#31 lore raw

PATCH v2 2/3 object-file: emit corruption errors when detected

In reply to Jonathan Tan

Instead of relying on errno being preserved across function calls, teach do_oid_object_info_extended() to itself report object corruption when it first detects it. There are 3 types of corruption being detected:
- when a replacement object is missing
- when a loose object is corrupt
- when a packed object is corrupt and the object cannot be read in another way

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

Besides being more resilient, this also prepares for a future patch in which an indirect caller of do_oid_object_info_extended() will need such functionality.

---
 object-file.c  | 63 ++++++++++++++++++++++++++------------------------
 object-store.h |  3 +++
 2 files changed, 36 insertions(+), 30 deletions(-)
object-file.c +33 -30
index 596dd049fd..c7a513d123 100644
@@ -1215,7 +1215,8 @@ static int quick_has_loose(struct repository *r,
  * searching for a loose object named "oid".
  */
 static void *map_loose_object_1(struct repository *r, const char *path,
-			     const struct object_id *oid, unsigned long *size)
+				const struct object_id *oid, unsigned long *size,
+				char **mapped_path)
 {
 	void *map;
 	int fd;
@@ -1224,6 +1225,9 @@ static void *map_loose_object_1(struct repository *r, const char *path,
 		fd = git_open(path);
 	else
 		fd = open_loose_object(r, oid, &path);
+	if (mapped_path)
+		*mapped_path = xstrdup(path);
+
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
@@ -1247,7 +1251,7 @@ void *map_loose_object(struct repository *r,
 		       const struct object_id *oid,
 		       unsigned long *size)
 {
-	return map_loose_object_1(r, NULL, oid, size);
+	return map_loose_object_1(r, NULL, oid, size, NULL);
 }
 
 enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
@@ -1428,6 +1432,7 @@ static int loose_object_info(struct repository *r,
 {
 	int status = 0;
 	unsigned long mapsize;
+	char *mapped_path = NULL;
 	void *map;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
@@ -1459,9 +1464,11 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object(r, oid, &mapsize);
-	if (!map)
+	map = map_loose_object_1(r, NULL, oid, &mapsize, &mapped_path);
+	if (!map) {
+		free(mapped_path);
 		return -1;
+	}
 
 	if (!oi->sizep)
 		oi->sizep = &size_scratch;
@@ -1497,8 +1504,13 @@ static int loose_object_info(struct repository *r,
 		break;
 	}
 
+	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
+		die(_("loose object %s (stored in %s) is corrupt"),
+		    oid_to_hex(oid), mapped_path);
+
 	git_inflate_end(&stream);
 cleanup:
+	free(mapped_path);
 	munmap(map, mapsize);
 	if (oi->sizep == &size_scratch)
 		oi->sizep = NULL;
@@ -1608,6 +1620,15 @@ static int do_oid_object_info_extended(struct repository *r,
 			continue;
 		}
 
+		if (flags & OBJECT_INFO_DIE_IF_CORRUPT) {
+			const struct packed_git *p;
+			if ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))
+				die(_("replacement %s not found for %s"),
+				    oid_to_hex(real), oid_to_hex(oid));
+			if ((p = has_packed_and_bad(r, real)))
+				die(_("packed object %s (stored in %s) is corrupt"),
+				    oid_to_hex(real), p->pack_name);
+		}
 		return -1;
 	}
 
@@ -1660,7 +1681,8 @@ int oid_object_info(struct repository *r,
 
 static void *read_object(struct repository *r,
 			 const struct object_id *oid, enum object_type *type,
-			 unsigned long *size)
+			 unsigned long *size,
+			 int die_if_corrupt)
 {
 	struct object_info oi = OBJECT_INFO_INIT;
 	void *content;
@@ -1668,7 +1690,9 @@ static void *read_object(struct repository *r,
 	oi.sizep = size;
 	oi.contentp = &content;
 
-	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
+	if (oid_object_info_extended(r, oid, &oi,
+				     die_if_corrupt ? OBJECT_INFO_DIE_IF_CORRUPT : 0)
+	    < 0)
 		return NULL;
 	return content;
 }
@@ -1704,35 +1728,14 @@ void *read_object_file_extended(struct repository *r,
 				int lookup_replace)
 {
 	void *data;
-	const struct packed_git *p;
-	const char *path;
-	struct stat st;
 	const struct object_id *repl = lookup_replace ?
 		lookup_replace_object(r, oid) : oid;
 
 	errno = 0;
-	data = read_object(r, repl, type, size);
+	data = read_object(r, repl, type, size, 1);
 	if (data)
 		return data;
 
-	obj_read_lock();
-	if (errno && errno != ENOENT)
-		die_errno(_("failed to read object %s"), oid_to_hex(oid));
-
-	/* die if we replaced an object with one that does not exist */
-	if (repl != oid)
-		die(_("replacement %s not found for %s"),
-		    oid_to_hex(repl), oid_to_hex(oid));
-
-	if (!stat_loose_object(r, repl, &st, &path))
-		die(_("loose object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), path);
-
-	if ((p = has_packed_and_bad(r, repl)))
-		die(_("packed object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), p->pack_name);
-	obj_read_unlock();
-
 	return NULL;
 }
 
@@ -2275,7 +2278,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 
 	if (has_loose_object(oid))
 		return 0;
-	buf = read_object(the_repository, oid, &type, &len);
+	buf = read_object(the_repository, oid, &type, &len, 0);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
 	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
@@ -2797,7 +2800,7 @@ int read_loose_object(const char *path,
 	char hdr[MAX_HEADER_LEN];
 	unsigned long *size = oi->sizep;
 
-	map = map_loose_object_1(the_repository, path, NULL, &mapsize);
+	map = map_loose_object_1(the_repository, path, NULL, &mapsize, NULL);
 	if (!map) {
 		error_errno(_("unable to mmap %s"), path);
 		goto out;
object-store.h +3 -0
index 1be57abaf1..01134ab5ec 100644
@@ -447,6 +447,9 @@ struct object_info {
  */
 #define OBJECT_INFO_FOR_PREFETCH (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)
 
+/* Die if object corruption (not just an object being missing) was detected. */
+#define OBJECT_INFO_DIE_IF_CORRUPT 64
+
 int oid_object_info_extended(struct repository *r,
 			     const struct object_id *,
 			     struct object_info *, unsigned flags);
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
#40 lore raw

PATCH v2 3/3 commit: don't lazy-fetch commits

In reply to Jonathan Tan

When parsing commits, fail fast when the commit is missing or corrupt, instead of attempting to fetch them. This is done by inlining repo_read_object_file() and setting the flag that prevents fetching.

This is motivated by a situation in which through a bug (not necessarily through Git), there was corruption in the object store of a partial clone. In this particular case, the problem was exposed when "git gc" tried to expire reflogs, which calls repo_parse_commit(), which triggers fetches of the missing commits.

(There are other possible solutions to this problem including passing an argument from "git gc" to "git reflog" to inhibit all lazy fetches, but I think that this fix is at the wrong level - fixing "git reflog" means that this particular command works fine, or so we think (it will fail if it somehow needs to read a legitimately missing blob, say, a .gitmodules file), but fixing repo_parse_commit() will fix a whole class of bugs.)

---
 commit.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
commit.c +13 -2
index 572301b80a..a02723f06b 100644
@@ -508,6 +508,17 @@ int repo_parse_commit_internal(struct repository *r,
 	enum object_type type;
 	void *buffer;
 	unsigned long size;
+	struct object_info oi = {
+		.typep = &type,
+		.sizep = &size,
+		.contentp = &buffer,
+	};
+	/*
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	int flags = OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT |
+		OBJECT_INFO_DIE_IF_CORRUPT;
 	int ret;
 
 	if (!item)
@@ -516,8 +527,8 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	if (oid_object_info_extended(r, &item->object.oid, &oi, flags) < 0)
 		return quiet_on_missing ? -1 :
 			error("Could not read %s",
 			     oid_to_hex(&item->object.oid));
Signature · 1 line
2.39.0.rc0.267.gcb52ba06e7-goog
#27 lore raw

Re: [PATCH v2 1/3] object-file: don't exit early if skipping loose

In reply to Jonathan Tan
Quoted 32 lines · Jonathan Tan

Instead, also search the submodule object stores and promisor remotes.

This also centralizes what happens when the object is not found (the "return -1"), which is useful for a subsequent patch.

---
 object-file.c | 23 +++++++++++------------
 1 file changed, 11 insertions(+), 12 deletions(-)
object-file.c +2 -12
index 26290554bb..596dd049fd 100644
@@ -1575,18 +1575,17 @@ static int do_oid_object_info_extended(struct repository *r,
 		if (find_pack_entry(r, real, &e))
 			break;
 
-		if (flags & OBJECT_INFO_IGNORE_LOOSE)
-			return -1;
-
-		/* Most likely it's a loose object. */
-		if (!loose_object_info(r, real, oi, flags))
-			return 0;
-
-		/* Not a loose object; someone else may have just packed it. */
-		if (!(flags & OBJECT_INFO_QUICK)) {
-			reprepare_packed_git(r);
-			if (find_pack_entry(r, real, &e))
-				break;
+		if (!(flags & OBJECT_INFO_IGNORE_LOOSE)) {
+			/* Most likely it's a loose object. */

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1589

Quoted 10 lines
object-file.c +9 -0
+			if (!loose_object_info(r, real, oi, flags))
+				return 0;
+
+			/* Not a loose object; someone else may have just packed it. */
+			if (!(flags & OBJECT_INFO_QUICK)) {
+				reprepare_packed_git(r);
+				if (find_pack_entry(r, real, &e))
+					break;
+			}
 		}

Hmph, who passes IGNORE_LOOSE and why? Explaining the answer to that question would give us confidence why this change is safe.

If the reason IGNORE_LOOSE is set by the callers is because they are interested only in locally packed objects, then this change would break them because they end up triggering the lazy fetch in the updated code, no? Or do all callers that set IGNORE_LOOSE drop the fetch_if_missing global before calling us?

Thanks.

#32 lore raw

Re: [PATCH v2 2/3] object-file: emit corruption errors when detected

In reply to Jonathan Tan
Quoted 22 lines · Jonathan Tan

Instead of relying on errno being preserved across function calls, teach do_oid_object_info_extended() to itself report object corruption when it first detects it. There are 3 types of corruption being detected:
- when a replacement object is missing
- when a loose object is corrupt
- when a packed object is corrupt and the object cannot be read in another way

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

Besides being more resilient, this also prepares for a future patch in which an indirect caller of do_oid_object_info_extended() will need such functionality.

Quoted 5 lines
---
 object-file.c  | 63 ++++++++++++++++++++++++++------------------------
 object-store.h |  3 +++
 2 files changed, 36 insertions(+), 30 deletions(-)

The implementation looks very straight-forward. Nicely done.

#41 lore raw

Re: [PATCH v2 3/3] commit: don't lazy-fetch commits

In reply to Jonathan Tan

commit.c int repo_parse_commit_internal(struct repository *r, · line 521

Quoted 6 lines · Jonathan Tan
commit.c +6 -0
+	/*
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	int flags = OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT |
+		OBJECT_INFO_DIE_IF_CORRUPT;

That's a quite helpful comment.

#33 lore raw

Re: [PATCH v2 2/3] object-file: emit corruption errors when detected

In reply to Jonathan Tan
Quoted 43 lines · Jonathan Tan

Instead of relying on errno being preserved across function calls, teach do_oid_object_info_extended() to itself report object corruption when it first detects it. There are 3 types of corruption being detected:
- when a replacement object is missing
- when a loose object is corrupt
- when a packed object is corrupt and the object cannot be read in another way

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

Besides being more resilient, this also prepares for a future patch in which an indirect caller of do_oid_object_info_extended() will need such functionality.

---
 object-file.c  | 63 ++++++++++++++++++++++++++------------------------
 object-store.h |  3 +++
 2 files changed, 36 insertions(+), 30 deletions(-)
object-file.c +2 -1
index 596dd049fd..c7a513d123 100644
@@ -1215,7 +1215,8 @@ static int quick_has_loose(struct repository *r,
  * searching for a loose object named "oid".
  */
 static void *map_loose_object_1(struct repository *r, const char *path,
-			     const struct object_id *oid, unsigned long *size)
+				const struct object_id *oid, unsigned long *size,
+				char **mapped_path)
 {
 	void *map;
 	int fd;

object-file.c static void *map_loose_object_1(struct repository *r, const char *path, · line 1233

Quoted 9 lines
object-file.c +3 -0
@@ -1224,6 +1225,9 @@ static void *map_loose_object_1(struct repository *r, const char *path,
 		fd = git_open(path);
 	else
 		fd = open_loose_object(r, oid, &path);
+	if (mapped_path)
+		*mapped_path = xstrdup(path);
+
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;

I find this map_loose_object_1() function to be rather "busy". Part of it's the pre-image.

Callers at the end of this series are:

   1254:        return map_loose_object_1(r, NULL, oid, size, NULL);
   1467:        map = map_loose_object_1(r, NULL, oid, &mapsize, &mapped_path);
   2803:        map = map_loose_object_1(the_repository, path, NULL, &mapsize, NULL);

So, either we know the path already, and we pass it in, or we don't know the path, and may or may not be interested in what the path ends up being.

Which is why we pass in both a "path" and a "mapped_path".

Then, somewhat confusingly (maybe I'm the only one who finds this odd") the "path" variable itself does double-duty within the function. If we have a "path" already we leave it alone, but if we don't it's NULL, and then we write our new path to it.

We *might* then have a path already, *and* write to the "mapped_path", but in that case we'd be xstrdup() ing a string the user passed in. But this API use would make no sense.

So shouldn't we at least have a:

	if (path && mapped_path)
		BUG("either tell me the path, or ask me, not both!");

But I think it's better to just separate these concerns. Most of this refactoring is good, but I think this bit went a step too far, and as a result we now need to memory manage this "mapped_path". I.e. we'd get a "struct strbuf"'s "buf" before, but now loose_object_info() needs to have it xstrdup()'d, just to free() it again.

Isn't the below squashed in better? I.e. just always pass the "path", but maybe pass a "fd=0", in which case the function might need to git_open() it.

Then have map_loose_object() and loose_object_info() call open_loose_object(), and pass in the "path" and "fd".

object-file.c +14 -18
index c7a513d123e..24793e1b479 100644
@@ -1214,19 +1214,13 @@ static int quick_has_loose(struct repository *r,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "oid".
  */
-static void *map_loose_object_1(struct repository *r, const char *path,
-				const struct object_id *oid, unsigned long *size,
-				char **mapped_path)
+static void *map_loose_object_1(struct repository *r, const char *const path,
+				int fd, unsigned long *size)
 {
 	void *map;
-	int fd;
 
-	if (path)
+	if (!fd)
 		fd = git_open(path);
-	else
-		fd = open_loose_object(r, oid, &path);
-	if (mapped_path)
-		*mapped_path = xstrdup(path);
 
 	map = NULL;
 	if (fd >= 0) {
@@ -1251,7 +1245,10 @@ void *map_loose_object(struct repository *r,
 		       const struct object_id *oid,
 		       unsigned long *size)
 {
-	return map_loose_object_1(r, NULL, oid, size, NULL);
+	const char *path;
+	int fd = open_loose_object(r, oid, &path);
+
+	return map_loose_object_1(r, path,fd, size);
 }
 
 enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
@@ -1432,7 +1429,6 @@ static int loose_object_info(struct repository *r,
 {
 	int status = 0;
 	unsigned long mapsize;
-	char *mapped_path = NULL;
 	void *map;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
@@ -1440,6 +1436,8 @@ static int loose_object_info(struct repository *r,
 	unsigned long size_scratch;
 	enum object_type type_scratch;
 	int allow_unknown = flags & OBJECT_INFO_ALLOW_UNKNOWN_TYPE;
+	int fd;
+	const char *path;
 
 	if (oi->delta_base_oid)
 		oidclr(oi->delta_base_oid);
@@ -1464,11 +1462,10 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object_1(r, NULL, oid, &mapsize, &mapped_path);
-	if (!map) {
-		free(mapped_path);
+	fd = open_loose_object(r, oid, &path);
+	map = map_loose_object_1(r, path, fd, &mapsize);
+	if (!map)
 		return -1;
-	}
 
 	if (!oi->sizep)
 		oi->sizep = &size_scratch;
@@ -1506,11 +1503,10 @@ static int loose_object_info(struct repository *r,
 
 	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
 		die(_("loose object %s (stored in %s) is corrupt"),
-		    oid_to_hex(oid), mapped_path);
+		    oid_to_hex(oid), path);
 
 	git_inflate_end(&stream);
 cleanup:
-	free(mapped_path);
 	munmap(map, mapsize);
 	if (oi->sizep == &size_scratch)
 		oi->sizep = NULL;
@@ -2800,7 +2796,7 @@ int read_loose_object(const char *path,
 	char hdr[MAX_HEADER_LEN];
 	unsigned long *size = oi->sizep;
 
-	map = map_loose_object_1(the_repository, path, NULL, &mapsize, NULL);
+	map = map_loose_object_1(the_repository, path, 0, &mapsize);
 	if (!map) {
 		error_errno(_("unable to mmap %s"), path);
 		goto out;
#28 lore raw

Re: [PATCH v2 1/3] object-file: don't exit early if skipping loose

In reply to Junio C Hamano
Quoted 8 lines · Junio C Hamano

Hmph, who passes IGNORE_LOOSE and why? Explaining the answer to that question would give us confidence why this change is safe.

If the reason IGNORE_LOOSE is set by the callers is because they are interested only in locally packed objects, then this change would break them because they end up triggering the lazy fetch in the updated code, no? Or do all callers that set IGNORE_LOOSE drop the fetch_if_missing global before calling us?

I wondered who those callers might be, too, because it is such a weird thing for a caller to want to care about (usually we try to abstract the object database).

It looks like the only user went away in 97b2fa08b6 (fetch-pack: drop custom loose object cache, 2018-11-12). So I think we just want to drop it:

object-file.c +0 -3
index 26290554bb..cf724bc19b 100644
@@ -1575,9 +1575,6 @@ static int do_oid_object_info_extended(struct repository *r,
 		if (find_pack_entry(r, real, &e))
 			break;
 
-		if (flags & OBJECT_INFO_IGNORE_LOOSE)
-			return -1;
-
 		/* Most likely it's a loose object. */
 		if (!loose_object_info(r, real, oi, flags))
 			return 0;
object-store.h +0 -2
index 1be57abaf1..371629c1e1 100644
@@ -434,8 +434,6 @@ struct object_info {
 #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2
 /* Do not retry packed storage after checking packed and loose storage */
 #define OBJECT_INFO_QUICK 8
-/* Do not check loose object */
-#define OBJECT_INFO_IGNORE_LOOSE 16
 /*
  * Do not attempt to fetch the object if missing (even if fetch_is_missing is
  * nonzero).

We could also renumber the later flags to keep them compact, but I don't have a strong opinion there.

-Peff

#39 lore raw

Re: [PATCH v2 2/3] object-file: emit corruption errors when detected

In reply to Jonathan Tan
Quoted 9 lines · Jonathan Tan

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

You're right that this will not say "oops, the object is corrupted" when we get EMFILE, etc, which is what 3ba7a06552 was fixing. But I think after your patch, we would also never actually say "could not open $path: too many open descriptors". I don't think that kicked in reliably with the current code, but it seems like something we are losing in this patch.

I.e., I think you'd want to also complain when map_loose_object() returns anything but ENOENT. The errno value there is reliable-ish, though it might be worth spending the extra work to preserve it across the close() calls, just in case. Or if you split the open/mmap as Ævar suggested, then it becomes:

fd = open_loose_object(r, oid, &path);

  if (fd < 0) {
	if (errno != ENOENT)
		error_errno("unable to open loose object %s", path);
	return -1;

}

buf = map_loose_object_1(fd, path);

  if (!buf) {
	/* not much point in complaining here, as xmmap will die on
	 * error. In theory this could catch fstat() problems, but
	 * probably map_loose_object_1() itself should do so, since
	 * it already is special-casing empty files.
	 */
        close(fd);
	return -1;

}

Quoted 15 lines
object-file.c +2 -1
index 596dd049fd..c7a513d123 100644
@@ -1215,7 +1215,8 @@ static int quick_has_loose(struct repository *r,
  * searching for a loose object named "oid".
  */
 static void *map_loose_object_1(struct repository *r, const char *path,
-			     const struct object_id *oid, unsigned long *size)
+				const struct object_id *oid, unsigned long *size,
+				char **mapped_path)
 {
 	void *map;
 	int fd;

object-file.c static void *map_loose_object_1(struct repository *r, const char *path, · line 1230

Quoted 6 lines
object-file.c +3 -0
@@ -1224,6 +1225,9 @@ static void *map_loose_object_1(struct repository *r, const char *path,
 		fd = git_open(path);
 	else
 		fd = open_loose_object(r, oid, &path);
+	if (mapped_path)
+		*mapped_path = xstrdup(path);
+

This introduces an extra malloc/free in every object lookup, even in the success case where we don't even bother using the value. It's probably not really noticeable, but it kind of feels wrong. Especially when open_loose_object() already returns a pointer to long-ish term storage.

One solution is...

object-file.c static int loose_object_info(struct repository *r, · line 1510

Quoted 8 lines
object-file.c +4 -0
@@ -1497,8 +1504,13 @@ static int loose_object_info(struct repository *r,
 		break;
 	}
 
+	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
+		die(_("loose object %s (stored in %s) is corrupt"),
+		    oid_to_hex(oid), mapped_path);
+

...we could just say "loose object %s is corrupt", which is generally sufficient (outside of alternates, you can only have one copy anyway).

But if we want to retain it, we could just redo the lookup in the error path, which is what the existing code (that you're deleting) does, via stat_loose_object(). That's even more wasteful than an extra malloc/free, but you only pay the cost for a corrupted object. It's racy, of course, but probably not in a meaningful way in practice.

Alternatively, I like Ævar's suggestion to just split map_loose_object(). Then this code would naturally have the path, via calling open_loose_object() itself.

object-store.h struct object_info { · line 451

Quoted 10 lines
object-store.h +2 -0
index 1be57abaf1..01134ab5ec 100644
@@ -447,6 +447,9 @@ struct object_info {
  */
 #define OBJECT_INFO_FOR_PREFETCH (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)
 
+/* Die if object corruption (not just an object being missing) was detected. */
+#define OBJECT_INFO_DIE_IF_CORRUPT 64

I have a suspicion that the world would be a better place if these die() calls simply went away, in favor of returning -1 up the stack. But I'm OK leaving it as-is for the sake of trying not to do too many things at once (I probably wouldn't have even mentioned it, except that if we do want to end up there in the long run, we'd eventually have to rip out this new flag and its associated plumbing).

-Peff

#29 lore raw

Re: [PATCH v2 1/3] object-file: don't exit early if skipping loose

In reply to Jeff King
Quoted 3 lines · Jeff King

I wondered who those callers might be, too, because it is such a weird thing for a caller to want to care about (usually we try to abstract the object database).

Exactly.

Quoted 2 lines

It looks like the only user went away in 97b2fa08b6 (fetch-pack: drop custom loose object cache, 2018-11-12).

Nice, very nice.

Quoted 21 lines

So I think we just want to drop it:

object-file.c +0 -3
index 26290554bb..cf724bc19b 100644
@@ -1575,9 +1575,6 @@ static int do_oid_object_info_extended(struct repository *r,
 		if (find_pack_entry(r, real, &e))
 			break;
 
-		if (flags & OBJECT_INFO_IGNORE_LOOSE)
-			return -1;
-
 		/* Most likely it's a loose object. */
 		if (!loose_object_info(r, real, oi, flags))
 			return 0;
object-store.h +0 -0
index 1be57abaf1..371629c1e1 100644

object-store.h struct object_info { · line 439

Quoted 8 lines
object-store.h +0 -2
@@ -434,8 +434,6 @@ struct object_info {
 #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2
 /* Do not retry packed storage after checking packed and loose storage */
 #define OBJECT_INFO_QUICK 8
-/* Do not check loose object */
-#define OBJECT_INFO_IGNORE_LOOSE 16
 /*
  * Do not attempt to fetch the object if missing (even if fetch_is_missing is
  * nonzero).

This would make Jonathan's change a lot transparent and intuitive if it is based on it, I would think.

Thanks for digging.

#42 lore raw

Re: [PATCH v2 3/3] commit: don't lazy-fetch commits

In reply to Jonathan Tan

commit.c int repo_parse_commit_internal(struct repository *r, · line 531

Quoted 8 lines · Jonathan Tan
commit.c +2 -2
@@ -516,8 +527,8 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	if (oid_object_info_extended(r, &item->object.oid, &oi, flags) < 0)

Nice. And this swap-out is much more obviously correct in this version of the series because read_object_file_extended() is now clearly a thin wrapper around oid_object_info_extended(), after your patch 2.

I actually think it would be beneficial to do a bit more refactoring there to eliminate read_object() entirely (in favor of just having the two callers use oid_object_info_extended() directly), and having repo_read_object_extended() pass OBJECT_INFO_LOOKUP_REPLACE instead of doing its own lookup.

But as that's all orthogonal to your goal, I don't mind if we punt on it for now. We can do it later on top.

-Peff

#34 lore raw

Re: [PATCH v2 2/3] object-file: emit corruption errors when detected

In reply to Ævar Arnfjörð Bjarmason
Quoted 6 lines · Ævar Arnfjörð Bjarmason

Isn't the below squashed in better? I.e. just always pass the "path", but maybe pass a "fd=0", in which case the function might need to git_open() it.

Then have map_loose_object() and loose_object_info() call open_loose_object(), and pass in the "path" and "fd".

I like this direction, though I'd give a few small suggestions. One is to make it unconditional to pass in a valid "fd". These kind of magic sentinel values sometimes lead to confusion or bugs, and it's easy enough for the caller to use git_open() itself.

In fact, in the one caller who cares, it lets us produce a nicer error message:

object-file.c +6 -4
index 24793e1b47..7c2a85132b 100644
@@ -1219,9 +1219,6 @@ static void *map_loose_object_1(struct repository *r, const char *const path,
 {
 	void *map;
 
-	if (!fd)
-		fd = git_open(path);
-
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
@@ -2790,13 +2787,18 @@ int read_loose_object(const char *path,
 		      struct object_info *oi)
 {
 	int ret = -1;
+	int fd;
 	void *map = NULL;
 	unsigned long mapsize;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
 	unsigned long *size = oi->sizep;
 
-	map = map_loose_object_1(the_repository, path, 0, &mapsize);
+	fd = git_open(path);
+	if (fd < 0)
+		error_errno(_("unable to open %s"), path);
+
+	map = map_loose_object_1(the_repository, path, fd, &mapsize);
 	if (!map) {
 		error_errno(_("unable to mmap %s"), path);
 		goto out;

Review

Quoted 13 lines
(quoted diff) +3 -6
+static void *map_loose_object_1(struct repository *r, const char *const path,
+				int fd, unsigned long *size)
 {
 	void *map;
-	int fd;
 
-	if (path)
+	if (!fd)
 		fd = git_open(path);
-	else
-		fd = open_loose_object(r, oid, &path);
-	if (mapped_path)
-		*mapped_path = xstrdup(path);

The other weird thing here is ownership of "fd". Now some callers pass it in, but map_loose_object_1() always closes it. I think that's OK (since we want it closed even on success), but definitely surprising enough that we'd want to document that in a comment.

Review void *map_loose_object(struct repository *r, · line 1252

Quoted 10 lines
(quoted diff) +4 -1
@@ -1251,7 +1245,10 @@ void *map_loose_object(struct repository *r,
 		       const struct object_id *oid,
 		       unsigned long *size)
 {
-	return map_loose_object_1(r, NULL, oid, size, NULL);
+	const char *path;
+	int fd = open_loose_object(r, oid, &path);
+
+	return map_loose_object_1(r, path,fd, size);
 }

It's also kind of weird that map_loose_object_1() is a noop on a negative descriptor. That technically makes this correct, but I think it would be much less surprising to always take a valid descriptor, and this code should do:

if (fd)

	return -1;

return map_loose_object_1(r, path, fd, size);

If we are going to make map_loose_object_1() less confusing (and I think that is worth doing), let's go all the way.

-Peff

#35 lore raw

Re: [PATCH v2 2/3] object-file: emit corruption errors when detected

Quoted 41 lines · Jeff King
Quoted 6 lines · Ævar Arnfjörð Bjarmason

Isn't the below squashed in better? I.e. just always pass the "path", but maybe pass a "fd=0", in which case the function might need to git_open() it.

Then have map_loose_object() and loose_object_info() call open_loose_object(), and pass in the "path" and "fd".

I like this direction, though I'd give a few small suggestions. One is to make it unconditional to pass in a valid "fd". These kind of magic sentinel values sometimes lead to confusion or bugs, and it's easy enough for the caller to use git_open() itself.

In fact, in the one caller who cares, it lets us produce a nicer error message:

object-file.c +1 -3
index 24793e1b47..7c2a85132b 100644
@@ -1219,9 +1219,6 @@ static void *map_loose_object_1(struct repository *r, const char *const path,
 {
 	void *map;
 
-	if (!fd)
-		fd = git_open(path);
-
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
@@ -2790,13 +2787,18 @@ int read_loose_object(const char *path,
 		      struct object_info *oi)
 {
 	int ret = -1;
+	int fd;
 	void *map = NULL;
 	unsigned long mapsize;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
 	unsigned long *size = oi->sizep;

object-file.c int read_loose_object(const char *path, · line 2804

Quoted 10 lines
object-file.c +5 -1
 
-	map = map_loose_object_1(the_repository, path, 0, &mapsize);
+	fd = git_open(path);
+	if (fd < 0)
+		error_errno(_("unable to open %s"), path);
+
+	map = map_loose_object_1(the_repository, path, fd, &mapsize);
 	if (!map) {
 		error_errno(_("unable to mmap %s"), path);
 		goto out;

Yeah, I think that's even better, although...

Quoted 35 lines

Review

Quoted 13 lines
(quoted diff) +3 -6
+static void *map_loose_object_1(struct repository *r, const char *const path,
+				int fd, unsigned long *size)
 {
 	void *map;
-	int fd;
 
-	if (path)
+	if (!fd)
 		fd = git_open(path);
-	else
-		fd = open_loose_object(r, oid, &path);
-	if (mapped_path)
-		*mapped_path = xstrdup(path);

The other weird thing here is ownership of "fd". Now some callers pass it in, but map_loose_object_1() always closes it. I think that's OK (since we want it closed even on success), but definitely surprising enough that we'd want to document that in a comment.

Review void *map_loose_object(struct repository *r, · line 1252

Quoted 10 lines
(quoted diff) +4 -1
@@ -1251,7 +1245,10 @@ void *map_loose_object(struct repository *r,
 		       const struct object_id *oid,
 		       unsigned long *size)
 {
-	return map_loose_object_1(r, NULL, oid, size, NULL);
+	const char *path;
+	int fd = open_loose_object(r, oid, &path);
+
+	return map_loose_object_1(r, path,fd, size);
 }

It's also kind of weird that map_loose_object_1() is a noop on a negative descriptor. That technically makes this correct, but I think it would be much less surprising to always take a valid descriptor, and this code should do:

Quoted 6 lines

if (fd)

	return -1;

return map_loose_object_1(r, path, fd, size);

If we are going to make map_loose_object_1() less confusing (and I think that is worth doing), let's go all the way.

...maybe we should go further in the other direction. I.e. with my earlier suggestion we're left with the mess that the "fd" ownership isn't clear.

But what I was trying to do was fix up the ownership around the "mapped_path", but we don't need to xstrdup() it in the first place. We already have the caller of open_loose_object() not doing that, we can just say that you're not going to open two loose objects at a time.

Then this becomes easier, and we can just pass the maybe-NULL "const char **oid_path" all the way to open_loose_object():

object-file.c +21 -16
index c7a513d123e..6e900737b76 100644
@@ -1176,7 +1176,7 @@ static int stat_loose_object(struct repository *r, const struct object_id *oid,
  * descriptor. See the caveats on the "path" parameter above.
  */
 static int open_loose_object(struct repository *r,
-			     const struct object_id *oid, const char **path)
+			     const struct object_id *oid, const char **oid_path)
 {
 	int fd;
 	struct object_directory *odb;
@@ -1185,8 +1185,12 @@ static int open_loose_object(struct repository *r,
 
 	prepare_alt_odb(r);
 	for (odb = r->objects->odb; odb; odb = odb->next) {
-		*path = odb_loose_path(odb, &buf, oid);
-		fd = git_open(*path);
+		const char *path;
+
+		path = odb_loose_path(odb, &buf, oid);
+		if (oid_path)
+			*oid_path = path;
+		fd = git_open(path);
 		if (fd >= 0)
 			return fd;
 
@@ -1214,19 +1218,22 @@ static int quick_has_loose(struct repository *r,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "oid".
  */
-static void *map_loose_object_1(struct repository *r, const char *path,
+static void *map_loose_object_1(struct repository *r, const char *const path,
 				const struct object_id *oid, unsigned long *size,
-				char **mapped_path)
+				const char **oid_path)
 {
 	void *map;
 	int fd;
 
+	if (path && oid_path)
+		BUG("don't tell me about the path, and ask me what it is!");
+	else if (!(path || oid))
+		BUG("must get an OID or a path!");
+
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_loose_object(r, oid, &path);
-	if (mapped_path)
-		*mapped_path = xstrdup(path);
+		fd = open_loose_object(r, oid, oid_path);
 
 	map = NULL;
 	if (fd >= 0) {
@@ -1236,7 +1243,8 @@ static void *map_loose_object_1(struct repository *r, const char *path,
 			*size = xsize_t(st.st_size);
 			if (!*size) {
 				/* mmap() is forbidden on empty files */
-				error(_("object file %s is empty"), path);
+				error(_("object file %s is empty"),
+				      path ? path : *oid_path);
 				close(fd);
 				return NULL;
 			}
@@ -1432,7 +1440,7 @@ static int loose_object_info(struct repository *r,
 {
 	int status = 0;
 	unsigned long mapsize;
-	char *mapped_path = NULL;
+	const char *oid_path;
 	void *map;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
@@ -1464,11 +1472,9 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object_1(r, NULL, oid, &mapsize, &mapped_path);
-	if (!map) {
-		free(mapped_path);
+	map = map_loose_object_1(r, NULL, oid, &mapsize, &oid_path);
+	if (!map)
 		return -1;
-	}
 
 	if (!oi->sizep)
 		oi->sizep = &size_scratch;
@@ -1506,11 +1512,10 @@ static int loose_object_info(struct repository *r,
 
 	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
 		die(_("loose object %s (stored in %s) is corrupt"),
-		    oid_to_hex(oid), mapped_path);
+		    oid_to_hex(oid), oid_path);
 
 	git_inflate_end(&stream);
 cleanup:
-	free(mapped_path);
 	munmap(map, mapsize);
 	if (oi->sizep == &size_scratch)
 		oi->sizep = NULL;
#30 lore raw

Re: [PATCH v2 1/3] object-file: don't exit early if skipping loose

In reply to Junio C Hamano
Quoted 6 lines · Junio C Hamano
Quoted 2 lines

It looks like the only user went away in 97b2fa08b6 (fetch-pack: drop custom loose object cache, 2018-11-12).

Nice, very nice.

Quoted 1 line

So I think we just want to drop it:

[snip]

Quoted 4 lines

This would make Jonathan's change a lot transparent and intuitive if it is based on it, I would think.

Thanks for digging.

Ah, thanks for finding this. I'll make this change.

#36 lore raw

Re: [PATCH v2 2/3] object-file: emit corruption errors when detected

In reply to Ævar Arnfjörð Bjarmason
Quoted 1 line · Ævar Arnfjörð Bjarmason

Yeah, I think that's even better, although...

[snip]

Quoted 9 lines
Quoted 9 lines

It's also kind of weird that map_loose_object_1() is a noop on a negative descriptor. That technically makes this correct, but I think it would be much less surprising to always take a valid descriptor, and this code should do:

if (fd)

	return -1;

return map_loose_object_1(r, path, fd, size);

Quoted 6 lines
Quoted 2 lines

If we are going to make map_loose_object_1() less confusing (and I think that is worth doing), let's go all the way.

...maybe we should go further in the other direction. I.e. with my earlier suggestion we're left with the mess that the "fd" ownership isn't clear.

With Peff's suggestion I think we can make the function always close the fd. If we find it not to be clear, we can rename the function to ..._close_fd() or something like that.

Thanks to both of you for your suggestions.

#37 lore raw

Re: [PATCH v2 2/3] object-file: emit corruption errors when detected

Quoted 22 lines · Jonathan Tan
Quoted 1 line · Ævar Arnfjörð Bjarmason

Yeah, I think that's even better, although...

[snip]

Quoted 9 lines
Quoted 9 lines

It's also kind of weird that map_loose_object_1() is a noop on a negative descriptor. That technically makes this correct, but I think it would be much less surprising to always take a valid descriptor, and this code should do:

if (fd)

	return -1;

return map_loose_object_1(r, path, fd, size);

Quoted 6 lines
Quoted 2 lines

If we are going to make map_loose_object_1() less confusing (and I think that is worth doing), let's go all the way.

...maybe we should go further in the other direction. I.e. with my earlier suggestion we're left with the mess that the "fd" ownership isn't clear.

Quoted 5 lines

With Peff's suggestion I think we can make the function always close the fd. If we find it not to be clear, we can rename the function to ..._close_fd() or something like that.

Thanks to both of you for your suggestions.

I think that was my suggestion.

Peff's on top of that was to never have it *open* the fd, but I'd left one caller doing that.

I.e. that the ownership would still be passed to it, but at least it would always be passed, and wouldn't be the mixed affair that my initial suggestion left it at.

I'll leave it to you to pick through this.

I have a mild preference for my latest suggestion as the ownership of all the variables seems cleanest in that iteration. I.e. we don't need to xstrdup(), and the "fd" is always contained within map_loose_object_1().

We still have the "sometimes a path, sometimes I make a path from an oid" semantics though, but that seems unavoidable.

#38 lore raw

Re: [PATCH v2 2/3] object-file: emit corruption errors when detected

Quoted 7 lines · Ævar Arnfjörð Bjarmason

I have a mild preference for my latest suggestion as the ownership of all the variables seems cleanest in that iteration. I.e. we don't need to xstrdup(), and the "fd" is always contained within map_loose_object_1().

We still have the "sometimes a path, sometimes I make a path from an oid" semantics though, but that seems unavoidable.

Of the two warts, I think "this function consume the fd" is less weird than the two path variables (one sometimes-in and one sometimes-out). If the fd thing is too ugly, we could have the function _not_ consume the fd, but I think that probably makes the callers worse.

At any rate, we can wait and see what Jonathan comes up with.

(As an aside, thank you Jonathan for dealing with some of this long-standing ugliness; it is not directly related to your goal, but I think it's adjacent enough to merit doing it as part of the series).

-Peff

#43 lore raw

v3 cover Don't lazy-fetch commits when parsing them

In reply to Jonathan Tan

Thanks everyone for your review. map_loose_object_1() definitely looks less "busy" than before after following your suggestions.

Jonathan Tan (4):
object-file: remove OBJECT_INFO_IGNORE_LOOSE
object-file: refactor map_loose_object_1()
object-file: emit corruption errors when detected
commit: don't lazy-fetch commits

 commit.c       |  15 ++++++-
 object-file.c  | 111 +++++++++++++++++++++++++------------------------
 object-store.h |   7 ++--
 3 files changed, 73 insertions(+), 60 deletions(-)

Range-diff against v2:

1:  9ad34a1dce < -:  ---------- object-file: don't exit early if skipping loose
-:  ---------- > 1:  be0b08cac2 object-file: remove OBJECT_INFO_IGNORE_LOOSE
-:  ---------- > 2:  7419e4ac70 object-file: refactor map_loose_object_1()
2:  9ddfff3585 ! 3:  7c9ed861e7 object-file: emit corruption errors when detected
    @@ Commit message
         Signed-off-by: Jonathan Tan <[email protected]>
     
      ## object-file.c ##
    -@@ object-file.c: static int quick_has_loose(struct repository *r,
    -  * searching for a loose object named "oid".
    -  */
    - static void *map_loose_object_1(struct repository *r, const char *path,
    --			     const struct object_id *oid, unsigned long *size)
    -+				const struct object_id *oid, unsigned long *size,
    -+				char **mapped_path)
    - {
    - 	void *map;
    - 	int fd;
    -@@ object-file.c: static void *map_loose_object_1(struct repository *r, const char *path,
    - 		fd = git_open(path);
    - 	else
    - 		fd = open_loose_object(r, oid, &path);
    -+	if (mapped_path)
    -+		*mapped_path = xstrdup(path);
    -+
    - 	map = NULL;
    - 	if (fd >= 0) {
    - 		struct stat st;
    -@@ object-file.c: void *map_loose_object(struct repository *r,
    - 		       const struct object_id *oid,
    - 		       unsigned long *size)
    - {
    --	return map_loose_object_1(r, NULL, oid, size);
    -+	return map_loose_object_1(r, NULL, oid, size, NULL);
    - }
    - 
    - enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
     @@ object-file.c: static int loose_object_info(struct repository *r,
      {
      	int status = 0;
      	unsigned long mapsize;
    -+	char *mapped_path = NULL;
    ++	const char *path = NULL;
      	void *map;
      	git_zstream stream;
      	char hdr[MAX_HEADER_LEN];
    @@ object-file.c: static int loose_object_info(struct repository *r,
      	}
      
     -	map = map_loose_object(r, oid, &mapsize);
    --	if (!map)
    -+	map = map_loose_object_1(r, NULL, oid, &mapsize, &mapped_path);
    -+	if (!map) {
    -+		free(mapped_path);
    ++	map = map_loose_object_1(r, oid, &mapsize, &path);
    + 	if (!map)
      		return -1;
    -+	}
      
    - 	if (!oi->sizep)
    - 		oi->sizep = &size_scratch;
     @@ object-file.c: static int loose_object_info(struct repository *r,
      		break;
      	}
      
     +	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
     +		die(_("loose object %s (stored in %s) is corrupt"),
    -+		    oid_to_hex(oid), mapped_path);
    ++		    oid_to_hex(oid), path);
     +
      	git_inflate_end(&stream);
      cleanup:
    -+	free(mapped_path);
      	munmap(map, mapsize);
    - 	if (oi->sizep == &size_scratch)
    - 		oi->sizep = NULL;
     @@ object-file.c: static int do_oid_object_info_extended(struct repository *r,
      			continue;
      		}
    @@ object-file.c: int force_object_loose(const struct object_id *oid, time_t mtime)
      	if (!buf)
      		return error(_("cannot read object for %s"), oid_to_hex(oid));
      	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
    -@@ object-file.c: int read_loose_object(const char *path,
    - 	char hdr[MAX_HEADER_LEN];
    - 	unsigned long *size = oi->sizep;
    - 
    --	map = map_loose_object_1(the_repository, path, NULL, &mapsize);
    -+	map = map_loose_object_1(the_repository, path, NULL, &mapsize, NULL);
    - 	if (!map) {
    - 		error_errno(_("unable to mmap %s"), path);
    - 		goto out;
     
      ## object-store.h ##
     @@ object-store.h: struct object_info {
    @@ object-store.h: struct object_info {
      #define OBJECT_INFO_FOR_PREFETCH (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)
      
     +/* Die if object corruption (not just an object being missing) was detected. */
    -+#define OBJECT_INFO_DIE_IF_CORRUPT 64
    ++#define OBJECT_INFO_DIE_IF_CORRUPT 32
     +
      int oid_object_info_extended(struct repository *r,
      			     const struct object_id *,

3: c5fe42deb0 = 4: 5924a5120b commit: don't lazy-fetch commits

Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#44 lore raw

PATCH v3 1/4 object-file: remove OBJECT_INFO_IGNORE_LOOSE

Its last user was removed in 97b2fa08b6 (fetch-pack: drop custom loose object cache, 2018-11-12), so we can remove it.

---
 object-file.c  | 3 ---
 object-store.h | 4 +---
 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -3
index 26290554bb..cf724bc19b 100644
@@ -1575,9 +1575,6 @@ static int do_oid_object_info_extended(struct repository *r,
 		if (find_pack_entry(r, real, &e))
 			break;
 
-		if (flags & OBJECT_INFO_IGNORE_LOOSE)
-			return -1;
-
 		/* Most likely it's a loose object. */
 		if (!loose_object_info(r, real, oi, flags))
 			return 0;
object-store.h +1 -3
index 1be57abaf1..b1ec0bde82 100644
@@ -434,13 +434,11 @@ struct object_info {
 #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2
 /* Do not retry packed storage after checking packed and loose storage */
 #define OBJECT_INFO_QUICK 8
-/* Do not check loose object */
-#define OBJECT_INFO_IGNORE_LOOSE 16
 /*
  * Do not attempt to fetch the object if missing (even if fetch_is_missing is
  * nonzero).
  */
-#define OBJECT_INFO_SKIP_FETCH_OBJECT 32
+#define OBJECT_INFO_SKIP_FETCH_OBJECT 16
 /*
  * This is meant for bulk prefetching of missing blobs in a partial
  * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#45 lore raw

PATCH v3 2/4 object-file: refactor map_loose_object_1()

In reply to Jonathan Tan

This function can do 3 things:
1. Gets an fd given a path
2. Simultaneously gets a path and fd given an OID
3. Memory maps an fd

Split this function up. Only one caller needs 1, so inline that. As for 2, a future patch will also need this functionality and, in addition, the calculated path, so extract this into a separate function with an out parameter for the path.

---
 object-file.c | 60 +++++++++++++++++++++++++++++----------------------
 1 file changed, 34 insertions(+), 26 deletions(-)
object-file.c +34 -26
index cf724bc19b..d99d05839f 100644
@@ -1211,43 +1211,48 @@ static int quick_has_loose(struct repository *r,
 }
 
 /*
- * Map the loose object at "path" if it is not NULL, or the path found by
- * searching for a loose object named "oid".
+ * Map and close the given loose object fd. The path argument is used for
+ * error reporting.
  */
-static void *map_loose_object_1(struct repository *r, const char *path,
-			     const struct object_id *oid, unsigned long *size)
+static void *map_fd(int fd, const char *path, unsigned long *size)
 {
-	void *map;
-	int fd;
-
-	if (path)
-		fd = git_open(path);
-	else
-		fd = open_loose_object(r, oid, &path);
-	map = NULL;
-	if (fd >= 0) {
-		struct stat st;
+	void *map = NULL;
+	struct stat st;
 
-		if (!fstat(fd, &st)) {
-			*size = xsize_t(st.st_size);
-			if (!*size) {
-				/* mmap() is forbidden on empty files */
-				error(_("object file %s is empty"), path);
-				close(fd);
-				return NULL;
-			}
-			map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
+	if (!fstat(fd, &st)) {
+		*size = xsize_t(st.st_size);
+		if (!*size) {
+			/* mmap() is forbidden on empty files */
+			error(_("object file %s is empty"), path);
+			close(fd);
+			return NULL;
 		}
-		close(fd);
+		map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
 	}
+	close(fd);
 	return map;
 }
 
+static void *map_loose_object_1(struct repository *r,
+				const struct object_id *oid,
+				unsigned long *size,
+				const char **path)
+{
+	const char *p;
+	int fd = open_loose_object(r, oid, &p);
+
+	if (fd < 0)
+		return NULL;
+	if (path)
+		*path = p;
+	return map_fd(fd, p, size);
+}
+
 void *map_loose_object(struct repository *r,
 		       const struct object_id *oid,
 		       unsigned long *size)
 {
-	return map_loose_object_1(r, NULL, oid, size);
+	return map_loose_object_1(r, oid, size, NULL);
 }
 
 enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
@@ -2789,13 +2794,16 @@ int read_loose_object(const char *path,
 		      struct object_info *oi)
 {
 	int ret = -1;
+	int fd;
 	void *map = NULL;
 	unsigned long mapsize;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
 	unsigned long *size = oi->sizep;
 
-	map = map_loose_object_1(the_repository, path, NULL, &mapsize);
+	fd = git_open(path);
+	if (fd >= 0)
+		map = map_fd(fd, path, &mapsize);
 	if (!map) {
 		error_errno(_("unable to mmap %s"), path);
 		goto out;
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#50 lore raw

PATCH v3 3/4 object-file: emit corruption errors when detected

In reply to Jonathan Tan

Instead of relying on errno being preserved across function calls, teach do_oid_object_info_extended() to itself report object corruption when it first detects it. There are 3 types of corruption being detected:
- when a replacement object is missing
- when a loose object is corrupt
- when a packed object is corrupt and the object cannot be read in another way

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

Besides being more resilient, this also prepares for a future patch in which an indirect caller of do_oid_object_info_extended() will need such functionality.

---
 object-file.c  | 48 ++++++++++++++++++++++--------------------------
 object-store.h |  3 +++
 2 files changed, 25 insertions(+), 26 deletions(-)
object-file.c +22 -26
index d99d05839f..f166065f32 100644
@@ -1433,6 +1433,7 @@ static int loose_object_info(struct repository *r,
 {
 	int status = 0;
 	unsigned long mapsize;
+	const char *path = NULL;
 	void *map;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
@@ -1464,7 +1465,7 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object(r, oid, &mapsize);
+	map = map_loose_object_1(r, oid, &mapsize, &path);
 	if (!map)
 		return -1;
 
@@ -1502,6 +1503,10 @@ static int loose_object_info(struct repository *r,
 		break;
 	}
 
+	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
+		die(_("loose object %s (stored in %s) is corrupt"),
+		    oid_to_hex(oid), path);
+
 	git_inflate_end(&stream);
 cleanup:
 	munmap(map, mapsize);
@@ -1611,6 +1616,15 @@ static int do_oid_object_info_extended(struct repository *r,
 			continue;
 		}
 
+		if (flags & OBJECT_INFO_DIE_IF_CORRUPT) {
+			const struct packed_git *p;
+			if ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))
+				die(_("replacement %s not found for %s"),
+				    oid_to_hex(real), oid_to_hex(oid));
+			if ((p = has_packed_and_bad(r, real)))
+				die(_("packed object %s (stored in %s) is corrupt"),
+				    oid_to_hex(real), p->pack_name);
+		}
 		return -1;
 	}
 
@@ -1663,7 +1677,8 @@ int oid_object_info(struct repository *r,
 
 static void *read_object(struct repository *r,
 			 const struct object_id *oid, enum object_type *type,
-			 unsigned long *size)
+			 unsigned long *size,
+			 int die_if_corrupt)
 {
 	struct object_info oi = OBJECT_INFO_INIT;
 	void *content;
@@ -1671,7 +1686,9 @@ static void *read_object(struct repository *r,
 	oi.sizep = size;
 	oi.contentp = &content;
 
-	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
+	if (oid_object_info_extended(r, oid, &oi,
+				     die_if_corrupt ? OBJECT_INFO_DIE_IF_CORRUPT : 0)
+	    < 0)
 		return NULL;
 	return content;
 }
@@ -1707,35 +1724,14 @@ void *read_object_file_extended(struct repository *r,
 				int lookup_replace)
 {
 	void *data;
-	const struct packed_git *p;
-	const char *path;
-	struct stat st;
 	const struct object_id *repl = lookup_replace ?
 		lookup_replace_object(r, oid) : oid;
 
 	errno = 0;
-	data = read_object(r, repl, type, size);
+	data = read_object(r, repl, type, size, 1);
 	if (data)
 		return data;
 
-	obj_read_lock();
-	if (errno && errno != ENOENT)
-		die_errno(_("failed to read object %s"), oid_to_hex(oid));
-
-	/* die if we replaced an object with one that does not exist */
-	if (repl != oid)
-		die(_("replacement %s not found for %s"),
-		    oid_to_hex(repl), oid_to_hex(oid));
-
-	if (!stat_loose_object(r, repl, &st, &path))
-		die(_("loose object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), path);
-
-	if ((p = has_packed_and_bad(r, repl)))
-		die(_("packed object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), p->pack_name);
-	obj_read_unlock();
-
 	return NULL;
 }
 
@@ -2278,7 +2274,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 
 	if (has_loose_object(oid))
 		return 0;
-	buf = read_object(the_repository, oid, &type, &len);
+	buf = read_object(the_repository, oid, &type, &len, 0);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
 	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
object-store.h +3 -0
index b1ec0bde82..98c1d67946 100644
@@ -445,6 +445,9 @@ struct object_info {
  */
 #define OBJECT_INFO_FOR_PREFETCH (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)
 
+/* Die if object corruption (not just an object being missing) was detected. */
+#define OBJECT_INFO_DIE_IF_CORRUPT 32
+
 int oid_object_info_extended(struct repository *r,
 			     const struct object_id *,
 			     struct object_info *, unsigned flags);
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#55 lore raw

PATCH v3 4/4 commit: don't lazy-fetch commits

In reply to Jonathan Tan

When parsing commits, fail fast when the commit is missing or corrupt, instead of attempting to fetch them. This is done by inlining repo_read_object_file() and setting the flag that prevents fetching.

This is motivated by a situation in which through a bug (not necessarily through Git), there was corruption in the object store of a partial clone. In this particular case, the problem was exposed when "git gc" tried to expire reflogs, which calls repo_parse_commit(), which triggers fetches of the missing commits.

(There are other possible solutions to this problem including passing an argument from "git gc" to "git reflog" to inhibit all lazy fetches, but I think that this fix is at the wrong level - fixing "git reflog" means that this particular command works fine, or so we think (it will fail if it somehow needs to read a legitimately missing blob, say, a .gitmodules file), but fixing repo_parse_commit() will fix a whole class of bugs.)

---
 commit.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
commit.c +13 -2
index 572301b80a..a02723f06b 100644
@@ -508,6 +508,17 @@ int repo_parse_commit_internal(struct repository *r,
 	enum object_type type;
 	void *buffer;
 	unsigned long size;
+	struct object_info oi = {
+		.typep = &type,
+		.sizep = &size,
+		.contentp = &buffer,
+	};
+	/*
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	int flags = OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT |
+		OBJECT_INFO_DIE_IF_CORRUPT;
 	int ret;
 
 	if (!item)
@@ -516,8 +527,8 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	if (oid_object_info_extended(r, &item->object.oid, &oi, flags) < 0)
 		return quiet_on_missing ? -1 :
 			error("Could not read %s",
 			     oid_to_hex(&item->object.oid));
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#51 lore raw

Re: [PATCH v3 3/4] object-file: emit corruption errors when detected

In reply to Jonathan Tan
Quoted 9 lines · Jonathan Tan

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

I think this version still has the small issue that we'll _only_ surface a generic error return in such a case, and never report EMFILE specifically. I.e., I think we'd still want something like this on top:

object-file.c +8 -1
index dc7665d6fa..36082bc991 100644
@@ -1422,6 +1422,7 @@ static int loose_object_info(struct repository *r,
 			     struct object_info *oi, int flags)
 {
 	int status = 0;
+	int fd;
 	unsigned long mapsize;
 	const char *path = NULL;
 	void *map;
@@ -1455,7 +1456,13 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object_1(r, oid, &mapsize, &path);
+	fd = open_loose_object(r, oid, &path);
+	if (fd < 0) {
+		if (errno != ENOENT)
+			error_errno(_("unable to open loose object %s"), path);
+		return -1;
+	}
+	map = map_fd(fd, path, &mapsize);
 	if (!map)
 		return -1;
 

Otherwise ENOENT and EMFILE are indistinguishable from the user's perspective. And one is normal and routine, but the other points to something the user probably needs to fix.

-Peff

#46 lore raw

Re: [PATCH v3 2/4] object-file: refactor map_loose_object_1()

In reply to Jonathan Tan
Quoted 9 lines · Jonathan Tan

This function can do 3 things:
1. Gets an fd given a path
2. Simultaneously gets a path and fd given an OID
3. Memory maps an fd

Split this function up. Only one caller needs 1, so inline that. As for 2, a future patch will also need this functionality and, in addition, the calculated path, so extract this into a separate function with an out parameter for the path.

This is moving in a good direction. I like the name "map_fd" for the helper. Being able to give it a useful name like that is a good clue that it is doing a more focused and understandable job than the generic map_loose_object_1(). :)

In fact...

Quoted 12 lines
(quoted diff) +12 -0
+static void *map_loose_object_1(struct repository *r,
+				const struct object_id *oid,
+				unsigned long *size,
+				const char **path)
+{
+	const char *p;
+	int fd = open_loose_object(r, oid, &p);
+
+	if (fd < 0)
+		return NULL;
+	if (path)
+		*path = p;

Review

Quoted 10 lines
(quoted diff) +4 -1
+	return map_fd(fd, p, size);
+}
+
 void *map_loose_object(struct repository *r,
 		       const struct object_id *oid,
 		       unsigned long *size)
 {
-	return map_loose_object_1(r, NULL, oid, size);
+	return map_loose_object_1(r, oid, size, NULL);
 }

If you take my suggestion on patch 3, then the only other caller of map_loose_object_1() goes away, and we can fold it all into one reasonably-named function:

object-file.c +3 -13
index d99d05839f..429e3a746d 100644
@@ -1233,28 +1233,18 @@ static void *map_fd(int fd, const char *path, unsigned long *size)
 	return map;
 }
 
-static void *map_loose_object_1(struct repository *r,
-				const struct object_id *oid,
-				unsigned long *size,
-				const char **path)
+void *map_loose_object(struct repository *r,
+		       const struct object_id *oid,
+		       unsigned long *size)
 {
 	const char *p;
 	int fd = open_loose_object(r, oid, &p);
 
 	if (fd < 0)
 		return NULL;
-	if (path)
-		*path = p;
 	return map_fd(fd, p, size);
 }
 
-void *map_loose_object(struct repository *r,
-		       const struct object_id *oid,
-		       unsigned long *size)
-{
-	return map_loose_object_1(r, oid, size, NULL);
-}
-
 enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
 						    unsigned char *map,
 						    unsigned long mapsize,

-Peff

#56 lore raw

Re: [PATCH v3 4/4] commit: don't lazy-fetch commits

In reply to Jonathan Tan
Quoted 23 lines · Jonathan Tan
commit.c +11 -0
index 572301b80a..a02723f06b 100644
@@ -508,6 +508,17 @@ int repo_parse_commit_internal(struct repository *r,
 	enum object_type type;
 	void *buffer;
 	unsigned long size;
+	struct object_info oi = {
+		.typep = &type,
+		.sizep = &size,
+		.contentp = &buffer,
+	};
+	/*
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	int flags = OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT |
+		OBJECT_INFO_DIE_IF_CORRUPT;
 	int ret;
 
 	if (!item)

commit.c int repo_parse_commit_internal(struct repository *r, · line 531

Quoted 7 lines
commit.c +2 -2
@@ -516,8 +527,8 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	if (oid_object_info_extended(r, &item->object.oid, &oi, flags) < 0)

Style: you're adding another \n here, usually we'd prefer it, but here the function already has all these checks bundled together without a \n, and this "if" is followed by one without it.

But then again those two "if"'s have to do with populating the "oi" and then reading out the "type", so it's probably fine & OK.

#53 lore raw

Re: [PATCH v3 3/4] object-file: emit corruption errors when detected

In reply to Jonathan Tan
Quoted 33 lines · Jonathan Tan

Instead of relying on errno being preserved across function calls, teach do_oid_object_info_extended() to itself report object corruption when it first detects it. There are 3 types of corruption being detected:
- when a replacement object is missing
- when a loose object is corrupt
- when a packed object is corrupt and the object cannot be read in another way

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

Besides being more resilient, this also prepares for a future patch in which an indirect caller of do_oid_object_info_extended() will need such functionality.

---
 object-file.c  | 48 ++++++++++++++++++++++--------------------------
 object-store.h |  3 +++
 2 files changed, 25 insertions(+), 26 deletions(-)
object-file.c +0 -0
index d99d05839f..f166065f32 100644

object-file.c static int loose_object_info(struct repository *r, · line 1436

Quoted 4 lines
object-file.c +1 -0
@@ -1433,6 +1433,7 @@ static int loose_object_info(struct repository *r,
 {
 	int status = 0;
 	unsigned long mapsize;
+	const char *path = NULL;

I think the NULL assignment here should either go, or it's incomplete.

Below you chechk the return value of "map", so let's either trust it to populate "path" if it's returning success (which it does), *or* not trust it, init this to NULL, and below add....

Quoted 11 lines
object-file.c +1 -1
 	void *map;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
@@ -1464,7 +1465,7 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object(r, oid, &mapsize);
+	map = map_loose_object_1(r, oid, &mapsize, &path);
 	if (!map)
 		return -1;

....

	if (!path)
		BUG("map_loose_object_1 should have given us a path");

But I don't think it's good to just hide the potential difference between the two, especially as....

object-file.c static int loose_object_info(struct repository *r, · line 1508

Quoted 7 lines
object-file.c +3 -0
@@ -1502,6 +1503,10 @@ static int loose_object_info(struct repository *r,
 		break;
 	}
 
+	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
+		die(_("loose object %s (stored in %s) is corrupt"),
+		    oid_to_hex(oid), path);

...I think this is leftover from a previous round (or maybe not, I didn't check) where we did a free(path), but here we'd end up segfaulting (not on glibc, but some platforms) if we have a NULL path.

So init-ing it didn't help us, but just helps to hide that potential (and much worse) bug.

I think this change should also remove the existing "const char *path" in this function from the "if"'d scope omitted in this context.

The C compiler won't care, but to the human reader it's easier to reason about not shadowing the variable now, for as it turns out no reason, as they're effectively independent.

object-file.c static int do_oid_object_info_extended(struct repository *r, · line 1620

Quoted 9 lines
object-file.c +2 -0
 	git_inflate_end(&stream);
 cleanup:
 	munmap(map, mapsize);
@@ -1611,6 +1616,15 @@ static int do_oid_object_info_extended(struct repository *r,
 			continue;
 		}
 
+		if (flags & OBJECT_INFO_DIE_IF_CORRUPT) {
+			const struct packed_git *p;

Nit: add an extra \n here, between decls and code.

object-file.c static int do_oid_object_info_extended(struct repository *r,

Quoted 7 lines
object-file.c +3 -1
-	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
+	if (oid_object_info_extended(r, oid, &oi,
+				     die_if_corrupt ? OBJECT_INFO_DIE_IF_CORRUPT : 0)
+	    < 0)
 		return NULL;
 	return content;
 }

This is a very odd coding style/wrapping, to not even end up with a line shorter than 79 characters. You can instead do:

	if (oid_object_info_extended(r, oid, &oi, die_if_corrupt
				     ? OBJECT_INFO_DIE_IF_CORRUPT : 0) < 0)

Which is in line with our usual style, and does wrap before 79 characters...

object-store.h struct object_info { · line 449

Quoted 10 lines
object-store.h +2 -0
index b1ec0bde82..98c1d67946 100644
@@ -445,6 +445,9 @@ struct object_info {
  */
 #define OBJECT_INFO_FOR_PREFETCH (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)
 
+/* Die if object corruption (not just an object being missing) was detected. */
+#define OBJECT_INFO_DIE_IF_CORRUPT 32

Personally I wouldn't mind a short cleanup step in this series to change these to 1<<0, 1<<1 etc., as we do for almost everything els.

I.e. in an earlier step you removed the "16", and changed that "32" to "16", now we're adding a "32" again.

I also notice that you didn't just add a "4" here, which is an existing gap, which turns out to be a leftover bit from your 9c8a294a1ae (sha1-file: remove OBJECT_INFO_SKIP_CACHED, 2020-01-02) ~2 years ago :)

Maybe it's too much, and we could do it later, but something like this as a first step:

object-file.c +3 -2
index 26290554bb4..48eff3850f5 100644
@@ -1534,7 +1534,8 @@ int fetch_if_missing = 1;
 
 static int do_oid_object_info_extended(struct repository *r,
 				       const struct object_id *oid,
-				       struct object_info *oi, unsigned flags)
+				       struct object_info *oi,
+				       enum object_info_flags flags)
 {
 	static struct object_info blank_oi = OBJECT_INFO_INIT;
 	struct cached_object *co;
@@ -1633,7 +1634,7 @@ static int do_oid_object_info_extended(struct repository *r,
 }
 
 int oid_object_info_extended(struct repository *r, const struct object_id *oid,
-			     struct object_info *oi, unsigned flags)
+			     struct object_info *oi, enum object_info_flags flags)
 {
 	int ret;
 	obj_read_lock();
object-store.h +22 -19

50 lines not on this page · raw message

#47 lore raw

Re: [PATCH v3 2/4] object-file: refactor map_loose_object_1()

In reply to Jeff King
Quoted 3 lines · Jeff King

If you take my suggestion on patch 3, then the only other caller of map_loose_object_1() goes away, and we can fold it all into one reasonably-named function:

Ah, that is true as of this patch, but patch 3 introduces another caller of this function. I tried to allude to it in the commit message, but if there is a clearer way to explain that, please let me know.

#52 lore raw

Re: [PATCH v3 3/4] object-file: emit corruption errors when detected

In reply to Jeff King
Quoted 3 lines · Jeff King

I think this version still has the small issue that we'll _only_ surface a generic error return in such a case, and never report EMFILE specifically. I.e., I think we'd still want something like this on top:

[snip]

OK, I'll do this. This also has the advantage of not using map_loose_object_1, so I'll be able to inline it in the previous patch.

#54 lore raw

Re: [PATCH v3 3/4] object-file: emit corruption errors when detected

In reply to Ævar Arnfjörð Bjarmason
Quoted 7 lines · Ævar Arnfjörð Bjarmason

Review static int loose_object_info(struct repository *r, · line 1436

Quoted 5 lines
(quoted diff) +1 -0
@@ -1433,6 +1433,7 @@ static int loose_object_info(struct repository *r,
 {
 	int status = 0;
 	unsigned long mapsize;
+	const char *path = NULL;

I think the NULL assignment here should either go, or it's incomplete.

[snip]

Quoted 2 lines

So init-ing it didn't help us, but just helps to hide that potential (and much worse) bug.

Good catch. I'll remove the assignment.

Quoted 6 lines

I think this change should also remove the existing "const char *path" in this function from the "if"'d scope omitted in this context.

The C compiler won't care, but to the human reader it's easier to reason about not shadowing the variable now, for as it turns out no reason, as they're effectively independent.

Makes sense.

Quoted 11 lines

Review static int do_oid_object_info_extended(struct repository *r, · line 1620

Quoted 9 lines
(quoted diff) +2 -0
 	git_inflate_end(&stream);
 cleanup:
 	munmap(map, mapsize);
@@ -1611,6 +1616,15 @@ static int do_oid_object_info_extended(struct repository *r,
 			continue;
 		}
 
+		if (flags & OBJECT_INFO_DIE_IF_CORRUPT) {
+			const struct packed_git *p;

Nit: add an extra \n here, between decls and code.

OK.

Quoted 11 lines

Review static int do_oid_object_info_extended(struct repository *r,

Quoted 7 lines
(quoted diff) +3 -1
-	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
+	if (oid_object_info_extended(r, oid, &oi,
+				     die_if_corrupt ? OBJECT_INFO_DIE_IF_CORRUPT : 0)
+	    < 0)
 		return NULL;
 	return content;
 }

This is a very odd coding style/wrapping, to not even end up with a line shorter than 79 characters. You can instead do:

Quoted 4 lines
	if (oid_object_info_extended(r, oid, &oi, die_if_corrupt
				     ? OBJECT_INFO_DIE_IF_CORRUPT : 0) < 0)

Which is in line with our usual style, and does wrap before 79 characters...

I didn't want to split the ternary expression, but OK, I'll follow your wrapping.

Quoted 14 lines

object-store.h struct object_info { · line 449

Quoted 10 lines
object-store.h +2 -0
index b1ec0bde82..98c1d67946 100644
@@ -445,6 +445,9 @@ struct object_info {
  */
 #define OBJECT_INFO_FOR_PREFETCH (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)
 
+/* Die if object corruption (not just an object being missing) was detected. */
+#define OBJECT_INFO_DIE_IF_CORRUPT 32

Personally I wouldn't mind a short cleanup step in this series to change these to 1<<0, 1<<1 etc., as we do for almost everything els.

Quoted 6 lines

I.e. in an earlier step you removed the "16", and changed that "32" to "16", now we're adding a "32" again.

I also notice that you didn't just add a "4" here, which is an existing gap, which turns out to be a leftover bit from your 9c8a294a1ae (sha1-file: remove OBJECT_INFO_SKIP_CACHED, 2020-01-02) ~2 years ago :)

Ah...I didn't notice the 4 missing. But I think the series has gone over enough iterations now that I'd rather leave this as-is and maybe change this in a future patch.

#48 lore raw

Re: [PATCH v3 2/4] object-file: refactor map_loose_object_1()

In reply to Jonathan Tan
Quoted 8 lines · Jonathan Tan
Quoted 3 lines · Jeff King

If you take my suggestion on patch 3, then the only other caller of map_loose_object_1() goes away, and we can fold it all into one reasonably-named function:

Ah, that is true as of this patch, but patch 3 introduces another caller of this function. I tried to allude to it in the commit message, but if there is a clearer way to explain that, please let me know.

Yes, that's the "other caller" I was referring to. :) Hopefully it is more clear after you read my comments on v3.

-Peff

#49 lore raw

Re: [PATCH v3 2/4] object-file: refactor map_loose_object_1()

Quoted 7 lines · Jeff King
Quoted 5 lines · Jonathan Tan
Quoted 3 lines · Jeff King

If you take my suggestion on patch 3, then the only other caller of map_loose_object_1() goes away, and we can fold it all into one reasonably-named function:

Quoted 6 lines
Quoted 3 lines

Ah, that is true as of this patch, but patch 3 introduces another caller of this function. I tried to allude to it in the commit message, but if there is a clearer way to explain that, please let me know.

Yes, that's the "other caller" I was referring to. :) Hopefully it is more clear after you read my comments on v3.

Er, that should be "...comments on patch 3", of course, not "v3".

-Peff

#57 lore raw

v4 cover Don't lazy-fetch commits when parsing them

In reply to Jonathan Tan

Thanks everyone for your comments. Here's a reroll.

Jonathan Tan (4):
object-file: remove OBJECT_INFO_IGNORE_LOOSE
object-file: refactor map_loose_object_1()
object-file: emit corruption errors when detected
commit: don't lazy-fetch commits

 commit.c       |  15 ++++++-
 object-file.c  | 108 ++++++++++++++++++++++++-------------------------
 object-store.h |   7 ++--
 3 files changed, 69 insertions(+), 61 deletions(-)

Range-diff against v3:

1:  be0b08cac2 = 1:  be0b08cac2 object-file: remove OBJECT_INFO_IGNORE_LOOSE
2:  7419e4ac70 ! 2:  4b2fb68743 object-file: refactor map_loose_object_1()

@@ Commit message
2. Simultaneously gets a path and fd given an OID
3. Memory maps an fd

- Split this function up. Only one caller needs 1, so inline that. As for
- 2, a future patch will also need this functionality and, in addition,
- the calculated path, so extract this into a separate function with an
- out parameter for the path.
+ Keep 3 (renaming the function accordingly) and inline 1 and 2 into their
+ respective callers.

@@ object-file.c: static int quick_has_loose(struct repository *r,
return map;
}

-+static void *map_loose_object_1(struct repository *r,
-+ const struct object_id *oid,
-+ unsigned long *size,
-+ const char **path)
-+{
+@@ object-file.c: void *map_loose_object(struct repository *r,
+ const struct object_id *oid,
+ unsigned long *size)
+ {
+- return map_loose_object_1(r, NULL, oid, size);
+ const char *p;
+ int fd = open_loose_object(r, oid, &p);
+
+ if (fd < 0)
+ return NULL;
-+ if (path)
-+ *path = p;
+ return map_fd(fd, p, size);
-+}
-+
- void *map_loose_object(struct repository *r,
- const struct object_id *oid,
- unsigned long *size)
- {
-- return map_loose_object_1(r, NULL, oid, size);
-+ return map_loose_object_1(r, oid, size, NULL);
}

enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
3: 7c9ed861e7 ! 3: 07d28db92c object-file: emit corruption errors when detected

    @@ Commit message
         which an indirect caller of do_oid_object_info_extended() will need
         such functionality.
     
    +    Helped-by: Jeff King <[email protected]>
         Signed-off-by: Jonathan Tan <[email protected]>
     
      ## object-file.c ##
     @@ object-file.c: static int loose_object_info(struct repository *r,
    + 			     struct object_info *oi, int flags)
      {
      	int status = 0;
    ++	int fd;
      	unsigned long mapsize;
    -+	const char *path = NULL;
    ++	const char *path;
      	void *map;
      	git_zstream stream;
      	char hdr[MAX_HEADER_LEN];
    +@@ object-file.c: static int loose_object_info(struct repository *r,
    + 	 * object even exists.
    + 	 */
    + 	if (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {
    +-		const char *path;
    + 		struct stat st;
    + 		if (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))
    + 			return quick_has_loose(r, oid) ? 0 : -1;
     @@ object-file.c: static int loose_object_info(struct repository *r,
      		return 0;
      	}
      
     -	map = map_loose_object(r, oid, &mapsize);
    -+	map = map_loose_object_1(r, oid, &mapsize, &path);
    ++	fd = open_loose_object(r, oid, &path);
    ++	if (fd < 0) {
    ++		if (errno != ENOENT)
    ++			error_errno(_("unable to open loose object %s"), path);
    ++		return -1;
    ++	}
    ++	map = map_fd(fd, path, &mapsize);
      	if (!map)
      		return -1;
      
    @@ object-file.c: static void *read_object(struct repository *r,
      	oi.contentp = &content;
      
     -	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
    -+	if (oid_object_info_extended(r, oid, &oi,
    -+				     die_if_corrupt ? OBJECT_INFO_DIE_IF_CORRUPT : 0)
    -+	    < 0)
    ++	if (oid_object_info_extended(r, oid, &oi, die_if_corrupt
    ++				     ? OBJECT_INFO_DIE_IF_CORRUPT : 0) < 0)
      		return NULL;
      	return content;
      }

4: 5924a5120b = 4: 1a0cd5b244 commit: don't lazy-fetch commits

Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#58 lore raw

PATCH v4 1/4 object-file: remove OBJECT_INFO_IGNORE_LOOSE

Its last user was removed in 97b2fa08b6 (fetch-pack: drop custom loose object cache, 2018-11-12), so we can remove it.

---
 object-file.c  | 3 ---
 object-store.h | 4 +---
 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -3
index 26290554bb..cf724bc19b 100644
@@ -1575,9 +1575,6 @@ static int do_oid_object_info_extended(struct repository *r,
 		if (find_pack_entry(r, real, &e))
 			break;
 
-		if (flags & OBJECT_INFO_IGNORE_LOOSE)
-			return -1;
-
 		/* Most likely it's a loose object. */
 		if (!loose_object_info(r, real, oi, flags))
 			return 0;
object-store.h +1 -3
index 1be57abaf1..b1ec0bde82 100644
@@ -434,13 +434,11 @@ struct object_info {
 #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2
 /* Do not retry packed storage after checking packed and loose storage */
 #define OBJECT_INFO_QUICK 8
-/* Do not check loose object */
-#define OBJECT_INFO_IGNORE_LOOSE 16
 /*
  * Do not attempt to fetch the object if missing (even if fetch_is_missing is
  * nonzero).
  */
-#define OBJECT_INFO_SKIP_FETCH_OBJECT 32
+#define OBJECT_INFO_SKIP_FETCH_OBJECT 16
 /*
  * This is meant for bulk prefetching of missing blobs in a partial
  * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#59 lore raw

PATCH v4 2/4 object-file: refactor map_loose_object_1()

In reply to Jonathan Tan

This function can do 3 things:
1. Gets an fd given a path
2. Simultaneously gets a path and fd given an OID
3. Memory maps an fd

Keep 3 (renaming the function accordingly) and inline 1 and 2 into their respective callers.

---
 object-file.c | 50 ++++++++++++++++++++++++--------------------------
 1 file changed, 24 insertions(+), 26 deletions(-)
object-file.c +24 -26
index cf724bc19b..429e3a746d 100644
@@ -1211,35 +1211,25 @@ static int quick_has_loose(struct repository *r,
 }
 
 /*
- * Map the loose object at "path" if it is not NULL, or the path found by
- * searching for a loose object named "oid".
+ * Map and close the given loose object fd. The path argument is used for
+ * error reporting.
  */
-static void *map_loose_object_1(struct repository *r, const char *path,
-			     const struct object_id *oid, unsigned long *size)
+static void *map_fd(int fd, const char *path, unsigned long *size)
 {
-	void *map;
-	int fd;
-
-	if (path)
-		fd = git_open(path);
-	else
-		fd = open_loose_object(r, oid, &path);
-	map = NULL;
-	if (fd >= 0) {
-		struct stat st;
+	void *map = NULL;
+	struct stat st;
 
-		if (!fstat(fd, &st)) {
-			*size = xsize_t(st.st_size);
-			if (!*size) {
-				/* mmap() is forbidden on empty files */
-				error(_("object file %s is empty"), path);
-				close(fd);
-				return NULL;
-			}
-			map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
+	if (!fstat(fd, &st)) {
+		*size = xsize_t(st.st_size);
+		if (!*size) {
+			/* mmap() is forbidden on empty files */
+			error(_("object file %s is empty"), path);
+			close(fd);
+			return NULL;
 		}
-		close(fd);
+		map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
 	}
+	close(fd);
 	return map;
 }
 
@@ -1247,7 +1237,12 @@ void *map_loose_object(struct repository *r,
 		       const struct object_id *oid,
 		       unsigned long *size)
 {
-	return map_loose_object_1(r, NULL, oid, size);
+	const char *p;
+	int fd = open_loose_object(r, oid, &p);
+
+	if (fd < 0)
+		return NULL;
+	return map_fd(fd, p, size);
 }
 
 enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
@@ -2789,13 +2784,16 @@ int read_loose_object(const char *path,
 		      struct object_info *oi)
 {
 	int ret = -1;
+	int fd;
 	void *map = NULL;
 	unsigned long mapsize;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
 	unsigned long *size = oi->sizep;
 
-	map = map_loose_object_1(the_repository, path, NULL, &mapsize);
+	fd = git_open(path);
+	if (fd >= 0)
+		map = map_fd(fd, path, &mapsize);
 	if (!map) {
 		error_errno(_("unable to mmap %s"), path);
 		goto out;
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#60 lore raw

PATCH v4 3/4 object-file: emit corruption errors when detected

In reply to Jonathan Tan

Instead of relying on errno being preserved across function calls, teach do_oid_object_info_extended() to itself report object corruption when it first detects it. There are 3 types of corruption being detected:
- when a replacement object is missing
- when a loose object is corrupt
- when a packed object is corrupt and the object cannot be read in another way

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

Besides being more resilient, this also prepares for a future patch in which an indirect caller of do_oid_object_info_extended() will need such functionality.

---
 object-file.c  | 55 +++++++++++++++++++++++++-------------------------
 object-store.h |  3 +++
 2 files changed, 31 insertions(+), 27 deletions(-)
object-file.c +28 -27
index 429e3a746d..2a0df39822 100644
@@ -1422,7 +1422,9 @@ static int loose_object_info(struct repository *r,
 			     struct object_info *oi, int flags)
 {
 	int status = 0;
+	int fd;
 	unsigned long mapsize;
+	const char *path;
 	void *map;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
@@ -1443,7 +1445,6 @@ static int loose_object_info(struct repository *r,
 	 * object even exists.
 	 */
 	if (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {
-		const char *path;
 		struct stat st;
 		if (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))
 			return quick_has_loose(r, oid) ? 0 : -1;
@@ -1454,7 +1455,13 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object(r, oid, &mapsize);
+	fd = open_loose_object(r, oid, &path);
+	if (fd < 0) {
+		if (errno != ENOENT)
+			error_errno(_("unable to open loose object %s"), path);
+		return -1;
+	}
+	map = map_fd(fd, path, &mapsize);
 	if (!map)
 		return -1;
 
@@ -1492,6 +1499,10 @@ static int loose_object_info(struct repository *r,
 		break;
 	}
 
+	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
+		die(_("loose object %s (stored in %s) is corrupt"),
+		    oid_to_hex(oid), path);
+
 	git_inflate_end(&stream);
 cleanup:
 	munmap(map, mapsize);
@@ -1601,6 +1612,15 @@ static int do_oid_object_info_extended(struct repository *r,
 			continue;
 		}
 
+		if (flags & OBJECT_INFO_DIE_IF_CORRUPT) {
+			const struct packed_git *p;
+			if ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))
+				die(_("replacement %s not found for %s"),
+				    oid_to_hex(real), oid_to_hex(oid));
+			if ((p = has_packed_and_bad(r, real)))
+				die(_("packed object %s (stored in %s) is corrupt"),
+				    oid_to_hex(real), p->pack_name);
+		}
 		return -1;
 	}
 
@@ -1653,7 +1673,8 @@ int oid_object_info(struct repository *r,
 
 static void *read_object(struct repository *r,
 			 const struct object_id *oid, enum object_type *type,
-			 unsigned long *size)
+			 unsigned long *size,
+			 int die_if_corrupt)
 {
 	struct object_info oi = OBJECT_INFO_INIT;
 	void *content;
@@ -1661,7 +1682,8 @@ static void *read_object(struct repository *r,
 	oi.sizep = size;
 	oi.contentp = &content;
 
-	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
+	if (oid_object_info_extended(r, oid, &oi, die_if_corrupt
+				     ? OBJECT_INFO_DIE_IF_CORRUPT : 0) < 0)
 		return NULL;
 	return content;
 }
@@ -1697,35 +1719,14 @@ void *read_object_file_extended(struct repository *r,
 				int lookup_replace)
 {
 	void *data;
-	const struct packed_git *p;
-	const char *path;
-	struct stat st;
 	const struct object_id *repl = lookup_replace ?
 		lookup_replace_object(r, oid) : oid;
 
 	errno = 0;
-	data = read_object(r, repl, type, size);
+	data = read_object(r, repl, type, size, 1);
 	if (data)
 		return data;
 
-	obj_read_lock();
-	if (errno && errno != ENOENT)
-		die_errno(_("failed to read object %s"), oid_to_hex(oid));
-
-	/* die if we replaced an object with one that does not exist */
-	if (repl != oid)
-		die(_("replacement %s not found for %s"),
-		    oid_to_hex(repl), oid_to_hex(oid));
-
-	if (!stat_loose_object(r, repl, &st, &path))
-		die(_("loose object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), path);
-
-	if ((p = has_packed_and_bad(r, repl)))
-		die(_("packed object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), p->pack_name);
-	obj_read_unlock();
-
 	return NULL;
 }
 
@@ -2268,7 +2269,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 
 	if (has_loose_object(oid))
 		return 0;
-	buf = read_object(the_repository, oid, &type, &len);
+	buf = read_object(the_repository, oid, &type, &len, 0);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
 	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
object-store.h +3 -0
index b1ec0bde82..98c1d67946 100644
@@ -445,6 +445,9 @@ struct object_info {
  */
 #define OBJECT_INFO_FOR_PREFETCH (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)
 
+/* Die if object corruption (not just an object being missing) was detected. */
+#define OBJECT_INFO_DIE_IF_CORRUPT 32
+
 int oid_object_info_extended(struct repository *r,
 			     const struct object_id *,
 			     struct object_info *, unsigned flags);
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#70 lore raw

PATCH v4 4/4 commit: don't lazy-fetch commits

In reply to Jonathan Tan

When parsing commits, fail fast when the commit is missing or corrupt, instead of attempting to fetch them. This is done by inlining repo_read_object_file() and setting the flag that prevents fetching.

This is motivated by a situation in which through a bug (not necessarily through Git), there was corruption in the object store of a partial clone. In this particular case, the problem was exposed when "git gc" tried to expire reflogs, which calls repo_parse_commit(), which triggers fetches of the missing commits.

(There are other possible solutions to this problem including passing an argument from "git gc" to "git reflog" to inhibit all lazy fetches, but I think that this fix is at the wrong level - fixing "git reflog" means that this particular command works fine, or so we think (it will fail if it somehow needs to read a legitimately missing blob, say, a .gitmodules file), but fixing repo_parse_commit() will fix a whole class of bugs.)

---
 commit.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
commit.c +13 -2
index 572301b80a..a02723f06b 100644
@@ -508,6 +508,17 @@ int repo_parse_commit_internal(struct repository *r,
 	enum object_type type;
 	void *buffer;
 	unsigned long size;
+	struct object_info oi = {
+		.typep = &type,
+		.sizep = &size,
+		.contentp = &buffer,
+	};
+	/*
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	int flags = OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT |
+		OBJECT_INFO_DIE_IF_CORRUPT;
 	int ret;
 
 	if (!item)
@@ -516,8 +527,8 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	if (oid_object_info_extended(r, &item->object.oid, &oi, flags) < 0)
 		return quiet_on_missing ? -1 :
 			error("Could not read %s",
 			     oid_to_hex(&item->object.oid));
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#61 lore raw

Re: [PATCH v4 3/4] object-file: emit corruption errors when detected

In reply to Jonathan Tan

object-file.c static int loose_object_info(struct repository *r, · line 1463

Quoted 6 lines · Jonathan Tan
object-file.c +6 -0
+	fd = open_loose_object(r, oid, &path);
+	if (fd < 0) {
+		if (errno != ENOENT)
+			error_errno(_("unable to open loose object %s"), path);
+		return -1;
+	}

I know there was a discussion in the previous round, but is this use of path truly safe? Currently it may happen to be as long as there is at least one element on the odb list, but when thinking things through with future-proofing point of view, I do not think assuming that path is always computable is a healthy thing to do in the longer term.

Our "struct object_id" may be extended in the future and allow us to express "invalid" object name, in which case the error return we get may not even be about "loose object file not openable" but "there will never be a loose object file for such an invalid object name", in which case there won't be any path returned from the function.

Other than that, the series looks quite clearly written. Nicely done.

Thanks.

#62 lore raw

Re: [PATCH v4 3/4] object-file: emit corruption errors when detected

Quoted 16 lines · Junio C Hamano

object-file.c static int loose_object_info(struct repository *r, · line 1463

Quoted 6 lines · Jonathan Tan
object-file.c +6 -0
+	fd = open_loose_object(r, oid, &path);
+	if (fd < 0) {
+		if (errno != ENOENT)
+			error_errno(_("unable to open loose object %s"), path);
+		return -1;
+	}

I know there was a discussion in the previous round, but is this use of path truly safe? Currently it may happen to be as long as there is at least one element on the odb list, but when thinking things through with future-proofing point of view, I do not think assuming that path is always computable is a healthy thing to do in the longer term.

Quoted 5 lines

Our "struct object_id" may be extended in the future and allow us to express "invalid" object name, in which case the error return we get may not even be about "loose object file not openable" but "there will never be a loose object file for such an invalid object name", in which case there won't be any path returned from the function.

Ah, good point. I think what I can do is to document the function to only return a path if a path was involved in the error somehow, and make anything that uses "path" in the caller check for NULL.

Quoted 4 lines

Other than that, the series looks quite clearly written. Nicely done.

Thanks.

Thanks for taking a look.

#63 lore raw

Re: [PATCH v4 3/4] object-file: emit corruption errors when detected

In reply to Junio C Hamano
Quoted 16 lines · Junio C Hamano

object-file.c static int loose_object_info(struct repository *r, · line 1463

Quoted 6 lines · Jonathan Tan
object-file.c +6 -0
+	fd = open_loose_object(r, oid, &path);
+	if (fd < 0) {
+		if (errno != ENOENT)
+			error_errno(_("unable to open loose object %s"), path);
+		return -1;
+	}

I know there was a discussion in the previous round, but is this use of path truly safe? Currently it may happen to be as long as there is at least one element on the odb list, but when thinking things through with future-proofing point of view, I do not think assuming that path is always computable is a healthy thing to do in the longer term.

Quoted 5 lines

Our "struct object_id" may be extended in the future and allow us to express "invalid" object name, in which case the error return we get may not even be about "loose object file not openable" but "there will never be a loose object file for such an invalid object name", in which case there won't be any path returned from the function.

Actually, I think it is much worse than that. The code as written above is already buggy (which is my fault, as I suggested it).

In open_loose_object() we'll continue to iterate and pick out the "most interesting errno". But we'll throw away the path that gave us that errno. So we might well say:

unable to open loose object /some/alternate/12/34abcd: permission denied

when the actual problem is in /main/objdir/12/34abcd.

It's fixable, but with some pain in handling the allocations. I think it would be sufficient to just say:

  error_errno(_("unable to open loose object %s"), oid_to_hex(oid));

here. And possibly put a comment above open_loose_object() that "path" is only guaranteed to point to something sensible when a non-negative value is returned.

-Peff

#64 lore raw

Re: [PATCH v4 3/4] object-file: emit corruption errors when detected

Quoted 9 lines · Jeff King

Actually, I think it is much worse than that. The code as written above is already buggy (which is my fault, as I suggested it).

In open_loose_object() we'll continue to iterate and pick out the "most interesting errno". But we'll throw away the path that gave us that errno. So we might well say:

unable to open loose object /some/alternate/12/34abcd: permission denied

Quoted 8 lines

when the actual problem is in /main/objdir/12/34abcd.

It's fixable, but with some pain in handling the allocations. I think it would be sufficient to just say:

  error_errno(_("unable to open loose object %s"), oid_to_hex(oid));

here.

OK, let's go with this.

Quoted 3 lines

And possibly put a comment above open_loose_object() that "path" is only guaranteed to point to something sensible when a non-negative value is returned.

Junio made a point that there could, for example, be no path when the odb list is empty (maybe in the future) so I don't think this would be sufficient. But there is already a comment there pointing to a comment in another function that states "path ... (if any)" so this is something that callers should already take care of. In my changes, I'll initialize it to NULL and whenever I use it, I'll check for non-NULL first.

#65 lore raw

Re: [PATCH v4 3/4] object-file: emit corruption errors when detected

Quoted 10 lines · Jonathan Tan
Quoted 3 lines

And possibly put a comment above open_loose_object() that "path" is only guaranteed to point to something sensible when a non-negative value is returned.

Junio made a point that there could, for example, be no path when the odb list is empty (maybe in the future) so I don't think this would be sufficient. But there is already a comment there pointing to a comment in another function that states "path ... (if any)" so this is something that callers should already take care of. In my changes, I'll initialize it to NULL and whenever I use it, I'll check for non-NULL first.

If we return a non-negative value, then we opened something, so by definition, don't we have a path of the thing we opened?

I think the case Junio mentioned was if we for some reason didn't look at _any_ path. In which case we'd be returning an error.

-Peff

#66 lore raw

Re: [PATCH v4 3/4] object-file: emit corruption errors when detected

Quoted 13 lines · Jeff King
Quoted 10 lines · Jonathan Tan
Quoted 3 lines

And possibly put a comment above open_loose_object() that "path" is only guaranteed to point to something sensible when a non-negative value is returned.

Junio made a point that there could, for example, be no path when the odb list is empty (maybe in the future) so I don't think this would be sufficient. But there is already a comment there pointing to a comment in another function that states "path ... (if any)" so this is something that callers should already take care of. In my changes, I'll initialize it to NULL and whenever I use it, I'll check for non-NULL first.

Quoted 5 lines

If we return a non-negative value, then we opened something, so by definition, don't we have a path of the thing we opened?

I think the case Junio mentioned was if we for some reason didn't look at _any_ path. In which case we'd be returning an error.

Ah, my reading comprehension is failing me, sorry. We do want "path" to point to something sensible (well, whenever we can) when an error occurs, though, since we want to include that path in our error message when DIE_IF_CORRUPT is used. So guaranteeing "path" when a non-negative value is returned (and hence, no error occurred) is not so useful.

#67 lore raw

Re: [PATCH v4 3/4] object-file: emit corruption errors when detected

Quoted 11 lines · Jonathan Tan
Quoted 5 lines

If we return a non-negative value, then we opened something, so by definition, don't we have a path of the thing we opened?

I think the case Junio mentioned was if we for some reason didn't look at _any_ path. In which case we'd be returning an error.

Ah, my reading comprehension is failing me, sorry. We do want "path" to point to something sensible (well, whenever we can) when an error occurs, though, since we want to include that path in our error message when DIE_IF_CORRUPT is used. So guaranteeing "path" when a non-negative value is returned (and hence, no error occurred) is not so useful.

But we only DIE_IF_CORRUPT when there is actual corruption, which means we've opened an object file, and "path" is valid.

The only time "path" would be invalid is if open_loose_object() itself returns an error, which is the message under discussion:

	fd = open_loose_object(r, oid, &path);
	if (fd < 0) {
		if (errno != ENOENT)
			error_errno(_("unable to open loose object %s"), path);
		return -1;
	}

If that stops expecting "path" to be valid (and just mentions the oid), then the rest of loose_object_info() should be fine.

-Peff

#71 lore raw

v5 cover Don't lazy-fetch commits when parsing them

In reply to Jonathan Tan

Thanks everyone for taking a look. Here's a reroll with safer path handling.

Jonathan Tan (4):
object-file: remove OBJECT_INFO_IGNORE_LOOSE
object-file: refactor map_loose_object_1()
object-file: emit corruption errors when detected
commit: don't lazy-fetch commits

 commit.c       |  15 ++++++-
 object-file.c  | 108 ++++++++++++++++++++++++-------------------------
 object-store.h |   7 ++--
 3 files changed, 69 insertions(+), 61 deletions(-)

Range-diff against v4:

1:  be0b08cac2 = 1:  be0b08cac2 object-file: remove OBJECT_INFO_IGNORE_LOOSE
2:  4b2fb68743 = 2:  4b2fb68743 object-file: refactor map_loose_object_1()
3:  07d28db92c ! 3:  a229ea0b11 object-file: emit corruption errors when detected
    @@ object-file.c: static int loose_object_info(struct repository *r,
      	int status = 0;
     +	int fd;
      	unsigned long mapsize;
    -+	const char *path;
    ++	const char *path = NULL;
      	void *map;
      	git_zstream stream;
      	char hdr[MAX_HEADER_LEN];
    @@ object-file.c: static int loose_object_info(struct repository *r,
     +	fd = open_loose_object(r, oid, &path);
     +	if (fd < 0) {
     +		if (errno != ENOENT)
    -+			error_errno(_("unable to open loose object %s"), path);
    ++			error_errno(_("unable to open loose object %s"), oid_to_hex(oid));
     +		return -1;
     +	}
     +	map = map_fd(fd, path, &mapsize);
    @@ object-file.c: static int loose_object_info(struct repository *r,
      		break;
      	}
      
    -+	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
    ++	if (status && path && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
     +		die(_("loose object %s (stored in %s) is corrupt"),
     +		    oid_to_hex(oid), path);
     +

4: 1a0cd5b244 = 4: b54972118a commit: don't lazy-fetch commits

Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#72 lore raw

PATCH v5 1/4 object-file: remove OBJECT_INFO_IGNORE_LOOSE

Its last user was removed in 97b2fa08b6 (fetch-pack: drop custom loose object cache, 2018-11-12), so we can remove it.

---
 object-file.c  | 3 ---
 object-store.h | 4 +---
 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -3
index 26290554bb..cf724bc19b 100644
@@ -1575,9 +1575,6 @@ static int do_oid_object_info_extended(struct repository *r,
 		if (find_pack_entry(r, real, &e))
 			break;
 
-		if (flags & OBJECT_INFO_IGNORE_LOOSE)
-			return -1;
-
 		/* Most likely it's a loose object. */
 		if (!loose_object_info(r, real, oi, flags))
 			return 0;
object-store.h +1 -3
index 1be57abaf1..b1ec0bde82 100644
@@ -434,13 +434,11 @@ struct object_info {
 #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2
 /* Do not retry packed storage after checking packed and loose storage */
 #define OBJECT_INFO_QUICK 8
-/* Do not check loose object */
-#define OBJECT_INFO_IGNORE_LOOSE 16
 /*
  * Do not attempt to fetch the object if missing (even if fetch_is_missing is
  * nonzero).
  */
-#define OBJECT_INFO_SKIP_FETCH_OBJECT 32
+#define OBJECT_INFO_SKIP_FETCH_OBJECT 16
 /*
  * This is meant for bulk prefetching of missing blobs in a partial
  * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#73 lore raw

PATCH v5 2/4 object-file: refactor map_loose_object_1()

In reply to Jonathan Tan

This function can do 3 things:
1. Gets an fd given a path
2. Simultaneously gets a path and fd given an OID
3. Memory maps an fd

Keep 3 (renaming the function accordingly) and inline 1 and 2 into their respective callers.

---
 object-file.c | 50 ++++++++++++++++++++++++--------------------------
 1 file changed, 24 insertions(+), 26 deletions(-)
object-file.c +24 -26
index cf724bc19b..429e3a746d 100644
@@ -1211,35 +1211,25 @@ static int quick_has_loose(struct repository *r,
 }
 
 /*
- * Map the loose object at "path" if it is not NULL, or the path found by
- * searching for a loose object named "oid".
+ * Map and close the given loose object fd. The path argument is used for
+ * error reporting.
  */
-static void *map_loose_object_1(struct repository *r, const char *path,
-			     const struct object_id *oid, unsigned long *size)
+static void *map_fd(int fd, const char *path, unsigned long *size)
 {
-	void *map;
-	int fd;
-
-	if (path)
-		fd = git_open(path);
-	else
-		fd = open_loose_object(r, oid, &path);
-	map = NULL;
-	if (fd >= 0) {
-		struct stat st;
+	void *map = NULL;
+	struct stat st;
 
-		if (!fstat(fd, &st)) {
-			*size = xsize_t(st.st_size);
-			if (!*size) {
-				/* mmap() is forbidden on empty files */
-				error(_("object file %s is empty"), path);
-				close(fd);
-				return NULL;
-			}
-			map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
+	if (!fstat(fd, &st)) {
+		*size = xsize_t(st.st_size);
+		if (!*size) {
+			/* mmap() is forbidden on empty files */
+			error(_("object file %s is empty"), path);
+			close(fd);
+			return NULL;
 		}
-		close(fd);
+		map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
 	}
+	close(fd);
 	return map;
 }
 
@@ -1247,7 +1237,12 @@ void *map_loose_object(struct repository *r,
 		       const struct object_id *oid,
 		       unsigned long *size)
 {
-	return map_loose_object_1(r, NULL, oid, size);
+	const char *p;
+	int fd = open_loose_object(r, oid, &p);
+
+	if (fd < 0)
+		return NULL;
+	return map_fd(fd, p, size);
 }
 
 enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
@@ -2789,13 +2784,16 @@ int read_loose_object(const char *path,
 		      struct object_info *oi)
 {
 	int ret = -1;
+	int fd;
 	void *map = NULL;
 	unsigned long mapsize;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
 	unsigned long *size = oi->sizep;
 
-	map = map_loose_object_1(the_repository, path, NULL, &mapsize);
+	fd = git_open(path);
+	if (fd >= 0)
+		map = map_fd(fd, path, &mapsize);
 	if (!map) {
 		error_errno(_("unable to mmap %s"), path);
 		goto out;
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#77 lore raw

PATCH v5 4/4 commit: don't lazy-fetch commits

In reply to Jonathan Tan

When parsing commits, fail fast when the commit is missing or corrupt, instead of attempting to fetch them. This is done by inlining repo_read_object_file() and setting the flag that prevents fetching.

This is motivated by a situation in which through a bug (not necessarily through Git), there was corruption in the object store of a partial clone. In this particular case, the problem was exposed when "git gc" tried to expire reflogs, which calls repo_parse_commit(), which triggers fetches of the missing commits.

(There are other possible solutions to this problem including passing an argument from "git gc" to "git reflog" to inhibit all lazy fetches, but I think that this fix is at the wrong level - fixing "git reflog" means that this particular command works fine, or so we think (it will fail if it somehow needs to read a legitimately missing blob, say, a .gitmodules file), but fixing repo_parse_commit() will fix a whole class of bugs.)

---
 commit.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
commit.c +13 -2
index 572301b80a..a02723f06b 100644
@@ -508,6 +508,17 @@ int repo_parse_commit_internal(struct repository *r,
 	enum object_type type;
 	void *buffer;
 	unsigned long size;
+	struct object_info oi = {
+		.typep = &type,
+		.sizep = &size,
+		.contentp = &buffer,
+	};
+	/*
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	int flags = OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT |
+		OBJECT_INFO_DIE_IF_CORRUPT;
 	int ret;
 
 	if (!item)
@@ -516,8 +527,8 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	if (oid_object_info_extended(r, &item->object.oid, &oi, flags) < 0)
 		return quiet_on_missing ? -1 :
 			error("Could not read %s",
 			     oid_to_hex(&item->object.oid));
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#74 lore raw

PATCH v5 3/4 object-file: emit corruption errors when detected

In reply to Jonathan Tan

Instead of relying on errno being preserved across function calls, teach do_oid_object_info_extended() to itself report object corruption when it first detects it. There are 3 types of corruption being detected:
- when a replacement object is missing
- when a loose object is corrupt
- when a packed object is corrupt and the object cannot be read in another way

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

Besides being more resilient, this also prepares for a future patch in which an indirect caller of do_oid_object_info_extended() will need such functionality.

---
 object-file.c  | 55 +++++++++++++++++++++++++-------------------------
 object-store.h |  3 +++
 2 files changed, 31 insertions(+), 27 deletions(-)
object-file.c +28 -27
index 429e3a746d..e0cef8b906 100644
@@ -1422,7 +1422,9 @@ static int loose_object_info(struct repository *r,
 			     struct object_info *oi, int flags)
 {
 	int status = 0;
+	int fd;
 	unsigned long mapsize;
+	const char *path = NULL;
 	void *map;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
@@ -1443,7 +1445,6 @@ static int loose_object_info(struct repository *r,
 	 * object even exists.
 	 */
 	if (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {
-		const char *path;
 		struct stat st;
 		if (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))
 			return quick_has_loose(r, oid) ? 0 : -1;
@@ -1454,7 +1455,13 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object(r, oid, &mapsize);
+	fd = open_loose_object(r, oid, &path);
+	if (fd < 0) {
+		if (errno != ENOENT)
+			error_errno(_("unable to open loose object %s"), oid_to_hex(oid));
+		return -1;
+	}
+	map = map_fd(fd, path, &mapsize);
 	if (!map)
 		return -1;
 
@@ -1492,6 +1499,10 @@ static int loose_object_info(struct repository *r,
 		break;
 	}
 
+	if (status && path && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
+		die(_("loose object %s (stored in %s) is corrupt"),
+		    oid_to_hex(oid), path);
+
 	git_inflate_end(&stream);
 cleanup:
 	munmap(map, mapsize);
@@ -1601,6 +1612,15 @@ static int do_oid_object_info_extended(struct repository *r,
 			continue;
 		}
 
+		if (flags & OBJECT_INFO_DIE_IF_CORRUPT) {
+			const struct packed_git *p;
+			if ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))
+				die(_("replacement %s not found for %s"),
+				    oid_to_hex(real), oid_to_hex(oid));
+			if ((p = has_packed_and_bad(r, real)))
+				die(_("packed object %s (stored in %s) is corrupt"),
+				    oid_to_hex(real), p->pack_name);
+		}
 		return -1;
 	}
 
@@ -1653,7 +1673,8 @@ int oid_object_info(struct repository *r,
 
 static void *read_object(struct repository *r,
 			 const struct object_id *oid, enum object_type *type,
-			 unsigned long *size)
+			 unsigned long *size,
+			 int die_if_corrupt)
 {
 	struct object_info oi = OBJECT_INFO_INIT;
 	void *content;
@@ -1661,7 +1682,8 @@ static void *read_object(struct repository *r,
 	oi.sizep = size;
 	oi.contentp = &content;
 
-	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
+	if (oid_object_info_extended(r, oid, &oi, die_if_corrupt
+				     ? OBJECT_INFO_DIE_IF_CORRUPT : 0) < 0)
 		return NULL;
 	return content;
 }
@@ -1697,35 +1719,14 @@ void *read_object_file_extended(struct repository *r,
 				int lookup_replace)
 {
 	void *data;
-	const struct packed_git *p;
-	const char *path;
-	struct stat st;
 	const struct object_id *repl = lookup_replace ?
 		lookup_replace_object(r, oid) : oid;
 
 	errno = 0;
-	data = read_object(r, repl, type, size);
+	data = read_object(r, repl, type, size, 1);
 	if (data)
 		return data;
 
-	obj_read_lock();
-	if (errno && errno != ENOENT)
-		die_errno(_("failed to read object %s"), oid_to_hex(oid));
-
-	/* die if we replaced an object with one that does not exist */
-	if (repl != oid)
-		die(_("replacement %s not found for %s"),
-		    oid_to_hex(repl), oid_to_hex(oid));
-
-	if (!stat_loose_object(r, repl, &st, &path))
-		die(_("loose object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), path);
-
-	if ((p = has_packed_and_bad(r, repl)))
-		die(_("packed object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), p->pack_name);
-	obj_read_unlock();
-
 	return NULL;
 }
 
@@ -2268,7 +2269,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 
 	if (has_loose_object(oid))
 		return 0;
-	buf = read_object(the_repository, oid, &type, &len);
+	buf = read_object(the_repository, oid, &type, &len, 0);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
 	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
object-store.h +3 -0
index b1ec0bde82..98c1d67946 100644
@@ -445,6 +445,9 @@ struct object_info {
  */
 #define OBJECT_INFO_FOR_PREFETCH (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)
 
+/* Die if object corruption (not just an object being missing) was detected. */
+#define OBJECT_INFO_DIE_IF_CORRUPT 32
+
 int oid_object_info_extended(struct repository *r,
 			     const struct object_id *,
 			     struct object_info *, unsigned flags);
Signature · 1 line
2.39.0.rc1.256.g54fd8350bd-goog
#68 lore raw

Re: [PATCH v4 3/4] object-file: emit corruption errors when detected

In reply to Jeff King
Quoted 13 lines · Jeff King
Quoted 10 lines · Jonathan Tan
Quoted 3 lines

And possibly put a comment above open_loose_object() that "path" is only guaranteed to point to something sensible when a non-negative value is returned.

Junio made a point that there could, for example, be no path when the odb list is empty (maybe in the future) so I don't think this would be sufficient. But there is already a comment there pointing to a comment in another function that states "path ... (if any)" so this is something that callers should already take care of. In my changes, I'll initialize it to NULL and whenever I use it, I'll check for non-NULL first.

Quoted 2 lines

If we return a non-negative value, then we opened something, so by definition, don't we have a path of the thing we opened?

Hmm...are you saying "path is guaranteed when there is no error; when there is an error, the caller must check"? If yes, I think we are in agreement. In any case, to make things more concrete, I've just sent a new version [1].

[1] https://lore.kernel.org/git/[email protected]/

#75 lore raw

Re: [PATCH v5 3/4] object-file: emit corruption errors when detected

In reply to Jonathan Tan

object-file.c static int loose_object_info(struct repository *r, · line 1427

Quoted 11 lines · Jonathan Tan
object-file.c +2 -0
index 429e3a746d..e0cef8b906 100644
@@ -1422,7 +1422,9 @@ static int loose_object_info(struct repository *r,
 			     struct object_info *oi, int flags)
 {
 	int status = 0;
+	int fd;
 	unsigned long mapsize;
+	const char *path = NULL;

It may be OK to leave this uninitialized, as long as the contract of open_loose_object() is that a successful opening will report the path to the loose object file that was opened. Because ...

object-file.c static int loose_object_info(struct repository *r, · line 1462

Quoted 10 lines
object-file.c +5 -1
@@ -1454,7 +1455,13 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object(r, oid, &mapsize);
+	fd = open_loose_object(r, oid, &path);
+	if (fd < 0) {
+		if (errno != ENOENT)
+			error_errno(_("unable to open loose object %s"), oid_to_hex(oid));
+		return -1;

... we no longer refer to "path" which may not be populated here, and ...

object-file.c static int loose_object_info(struct repository *r, · line 1463

Quoted 1 line
object-file.c +1 -0
+	}

... at this point, we know we successfully opened, and populated path.

object-file.c static int loose_object_info(struct repository *r, · line 1502

Quoted 9 lines
object-file.c +2 -0
+	map = map_fd(fd, path, &mapsize);
 	if (!map)
 		return -1;
 
@@ -1492,6 +1499,10 @@ static int loose_object_info(struct repository *r,
 		break;
 	}
 
+	if (status && path && (flags & OBJECT_INFO_DIE_IF_CORRUPT))

Also, here, "path" should be valid, as we have successfully opened the loose object file (otherwise we would have hit the early return that reports only the oid_to_hex(oid)).

object-file.c static int loose_object_info(struct repository *r, · line 1504

Quoted 2 lines
object-file.c +2 -0
+		die(_("loose object %s (stored in %s) is corrupt"),
+		    oid_to_hex(oid), path);

If we didn't have path and did not die, we'd end up ignoring DIE_IF_CORRUPT request and force the caller to handle non-zero status. But I do not think that should happen, because path would be valid here. No?

#69 lore raw

Re: [PATCH v4 3/4] object-file: emit corruption errors when detected

In reply to Jonathan Tan
Quoted 7 lines · Jonathan Tan
Quoted 7 lines
Quoted 6 lines

Junio made a point that there could, for example, be no path when the odb list is empty (maybe in the future) so I don't think this would be sufficient. But there is already a comment there pointing to a comment in another function that states "path ... (if any)" so this is something that callers should already take care of. In my changes, I'll initialize it to NULL and whenever I use it, I'll check for non-NULL first.

Quoted 7 lines
Quoted 2 lines

If we return a non-negative value, then we opened something, so by definition, don't we have a path of the thing we opened?

Hmm...are you saying "path is guaranteed when there is no error; when there is an error, the caller must check"? If yes, I think we are in agreement. In any case, to make things more concrete, I've just sent a new version [1].

Almost. I'm saying "path is guaranteed when there is no error; when there is an error, the value of path is meaningless and should not be looked at".

If you want to enforce that open_loose_object() sets "path" to NULL on error, and then say "the caller must check", that would be valid. But without that, even checking it for NULL is pointless (because you may see a path which got ENOENT, even though we got an interesting errno from an earlier path).

-Peff

#76 lore raw

Re: [PATCH v5 3/4] object-file: emit corruption errors when detected

In reply to Junio C Hamano
Quoted 17 lines · Junio C Hamano

object-file.c static int loose_object_info(struct repository *r, · line 1427

Quoted 11 lines · Jonathan Tan
object-file.c +2 -0
index 429e3a746d..e0cef8b906 100644
@@ -1422,7 +1422,9 @@ static int loose_object_info(struct repository *r,
 			     struct object_info *oi, int flags)
 {
 	int status = 0;
+	int fd;
 	unsigned long mapsize;
+	const char *path = NULL;

It may be OK to leave this uninitialized, as long as the contract of open_loose_object() is that a successful opening will report the path to the loose object file that was opened. Because ...

Yeah, I'd agree that this initialization can be left off (and that the NULL checks later in the function are not needed).

-Peff

#78 lore raw

v6 cover Don't lazy-fetch commits when parsing them

In reply to Jonathan Tan

Thanks everyone once again and sorry for the churn. Hopefully I got it right this time.

open_loose_object() is documented to return the path of the object we found, so I think we already have that covered (if we detect that an object is corrupt, it follows that we would already have found the object in the first place).

Jonathan Tan (4):
object-file: remove OBJECT_INFO_IGNORE_LOOSE
object-file: refactor map_loose_object_1()
object-file: emit corruption errors when detected
commit: don't lazy-fetch commits

 commit.c       |  15 ++++++-
 object-file.c  | 108 ++++++++++++++++++++++++-------------------------
 object-store.h |   7 ++--
 3 files changed, 69 insertions(+), 61 deletions(-)

Range-diff against v5:

1:  be0b08cac2 = 1:  be0b08cac2 object-file: remove OBJECT_INFO_IGNORE_LOOSE
2:  4b2fb68743 = 2:  4b2fb68743 object-file: refactor map_loose_object_1()
3:  a229ea0b11 ! 3:  811620909a object-file: emit corruption errors when detected
    @@ object-file.c: static int loose_object_info(struct repository *r,
      	int status = 0;
     +	int fd;
      	unsigned long mapsize;
    -+	const char *path = NULL;
    ++	const char *path;
      	void *map;
      	git_zstream stream;
      	char hdr[MAX_HEADER_LEN];
    @@ object-file.c: static int loose_object_info(struct repository *r,
      		break;
      	}
      
    -+	if (status && path && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
    ++	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
     +		die(_("loose object %s (stored in %s) is corrupt"),
     +		    oid_to_hex(oid), path);
     +

4: b54972118a = 4: 8acf1a29e7 commit: don't lazy-fetch commits

Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#79 lore raw

PATCH v6 1/4 object-file: remove OBJECT_INFO_IGNORE_LOOSE

Its last user was removed in 97b2fa08b6 (fetch-pack: drop custom loose object cache, 2018-11-12), so we can remove it.

---
 object-file.c  | 3 ---
 object-store.h | 4 +---
 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -3
index 26290554bb..cf724bc19b 100644
@@ -1575,9 +1575,6 @@ static int do_oid_object_info_extended(struct repository *r,
 		if (find_pack_entry(r, real, &e))
 			break;
 
-		if (flags & OBJECT_INFO_IGNORE_LOOSE)
-			return -1;
-
 		/* Most likely it's a loose object. */
 		if (!loose_object_info(r, real, oi, flags))
 			return 0;
object-store.h +1 -3
index 1be57abaf1..b1ec0bde82 100644
@@ -434,13 +434,11 @@ struct object_info {
 #define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2
 /* Do not retry packed storage after checking packed and loose storage */
 #define OBJECT_INFO_QUICK 8
-/* Do not check loose object */
-#define OBJECT_INFO_IGNORE_LOOSE 16
 /*
  * Do not attempt to fetch the object if missing (even if fetch_is_missing is
  * nonzero).
  */
-#define OBJECT_INFO_SKIP_FETCH_OBJECT 32
+#define OBJECT_INFO_SKIP_FETCH_OBJECT 16
 /*
  * This is meant for bulk prefetching of missing blobs in a partial
  * clone. Implies OBJECT_INFO_SKIP_FETCH_OBJECT and OBJECT_INFO_QUICK
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#80 lore raw

PATCH v6 2/4 object-file: refactor map_loose_object_1()

In reply to Jonathan Tan

This function can do 3 things:
1. Gets an fd given a path
2. Simultaneously gets a path and fd given an OID
3. Memory maps an fd

Keep 3 (renaming the function accordingly) and inline 1 and 2 into their respective callers.

---
 object-file.c | 50 ++++++++++++++++++++++++--------------------------
 1 file changed, 24 insertions(+), 26 deletions(-)
object-file.c +24 -26
index cf724bc19b..429e3a746d 100644
@@ -1211,35 +1211,25 @@ static int quick_has_loose(struct repository *r,
 }
 
 /*
- * Map the loose object at "path" if it is not NULL, or the path found by
- * searching for a loose object named "oid".
+ * Map and close the given loose object fd. The path argument is used for
+ * error reporting.
  */
-static void *map_loose_object_1(struct repository *r, const char *path,
-			     const struct object_id *oid, unsigned long *size)
+static void *map_fd(int fd, const char *path, unsigned long *size)
 {
-	void *map;
-	int fd;
-
-	if (path)
-		fd = git_open(path);
-	else
-		fd = open_loose_object(r, oid, &path);
-	map = NULL;
-	if (fd >= 0) {
-		struct stat st;
+	void *map = NULL;
+	struct stat st;
 
-		if (!fstat(fd, &st)) {
-			*size = xsize_t(st.st_size);
-			if (!*size) {
-				/* mmap() is forbidden on empty files */
-				error(_("object file %s is empty"), path);
-				close(fd);
-				return NULL;
-			}
-			map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
+	if (!fstat(fd, &st)) {
+		*size = xsize_t(st.st_size);
+		if (!*size) {
+			/* mmap() is forbidden on empty files */
+			error(_("object file %s is empty"), path);
+			close(fd);
+			return NULL;
 		}
-		close(fd);
+		map = xmmap(NULL, *size, PROT_READ, MAP_PRIVATE, fd, 0);
 	}
+	close(fd);
 	return map;
 }
 
@@ -1247,7 +1237,12 @@ void *map_loose_object(struct repository *r,
 		       const struct object_id *oid,
 		       unsigned long *size)
 {
-	return map_loose_object_1(r, NULL, oid, size);
+	const char *p;
+	int fd = open_loose_object(r, oid, &p);
+
+	if (fd < 0)
+		return NULL;
+	return map_fd(fd, p, size);
 }
 
 enum unpack_loose_header_result unpack_loose_header(git_zstream *stream,
@@ -2789,13 +2784,16 @@ int read_loose_object(const char *path,
 		      struct object_info *oi)
 {
 	int ret = -1;
+	int fd;
 	void *map = NULL;
 	unsigned long mapsize;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
 	unsigned long *size = oi->sizep;
 
-	map = map_loose_object_1(the_repository, path, NULL, &mapsize);
+	fd = git_open(path);
+	if (fd >= 0)
+		map = map_fd(fd, path, &mapsize);
 	if (!map) {
 		error_errno(_("unable to mmap %s"), path);
 		goto out;
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#81 lore raw

PATCH v6 3/4 object-file: emit corruption errors when detected

In reply to Jonathan Tan

Instead of relying on errno being preserved across function calls, teach do_oid_object_info_extended() to itself report object corruption when it first detects it. There are 3 types of corruption being detected:
- when a replacement object is missing
- when a loose object is corrupt
- when a packed object is corrupt and the object cannot be read in another way

Note that in the RHS of this patch's diff, a check for ENOENT that was introduced in 3ba7a06552 (A loose object is not corrupt if it cannot be read due to EMFILE, 2010-10-28) is also removed. The purpose of this check is to avoid a false report of corruption if the errno contains something like EMFILE (or anything that is not ENOENT), in which case a more generic report is presented. Because, as of this patch, we no longer rely on such a heuristic to determine corruption, but surface the error message at the point when we read something that we did not expect, this check is no longer necessary.

Besides being more resilient, this also prepares for a future patch in which an indirect caller of do_oid_object_info_extended() will need such functionality.

---
 object-file.c  | 55 +++++++++++++++++++++++++-------------------------
 object-store.h |  3 +++
 2 files changed, 31 insertions(+), 27 deletions(-)
object-file.c +28 -27
index 429e3a746d..e55697e378 100644
@@ -1422,7 +1422,9 @@ static int loose_object_info(struct repository *r,
 			     struct object_info *oi, int flags)
 {
 	int status = 0;
+	int fd;
 	unsigned long mapsize;
+	const char *path;
 	void *map;
 	git_zstream stream;
 	char hdr[MAX_HEADER_LEN];
@@ -1443,7 +1445,6 @@ static int loose_object_info(struct repository *r,
 	 * object even exists.
 	 */
 	if (!oi->typep && !oi->type_name && !oi->sizep && !oi->contentp) {
-		const char *path;
 		struct stat st;
 		if (!oi->disk_sizep && (flags & OBJECT_INFO_QUICK))
 			return quick_has_loose(r, oid) ? 0 : -1;
@@ -1454,7 +1455,13 @@ static int loose_object_info(struct repository *r,
 		return 0;
 	}
 
-	map = map_loose_object(r, oid, &mapsize);
+	fd = open_loose_object(r, oid, &path);
+	if (fd < 0) {
+		if (errno != ENOENT)
+			error_errno(_("unable to open loose object %s"), oid_to_hex(oid));
+		return -1;
+	}
+	map = map_fd(fd, path, &mapsize);
 	if (!map)
 		return -1;
 
@@ -1492,6 +1499,10 @@ static int loose_object_info(struct repository *r,
 		break;
 	}
 
+	if (status && (flags & OBJECT_INFO_DIE_IF_CORRUPT))
+		die(_("loose object %s (stored in %s) is corrupt"),
+		    oid_to_hex(oid), path);
+
 	git_inflate_end(&stream);
 cleanup:
 	munmap(map, mapsize);
@@ -1601,6 +1612,15 @@ static int do_oid_object_info_extended(struct repository *r,
 			continue;
 		}
 
+		if (flags & OBJECT_INFO_DIE_IF_CORRUPT) {
+			const struct packed_git *p;
+			if ((flags & OBJECT_INFO_LOOKUP_REPLACE) && !oideq(real, oid))
+				die(_("replacement %s not found for %s"),
+				    oid_to_hex(real), oid_to_hex(oid));
+			if ((p = has_packed_and_bad(r, real)))
+				die(_("packed object %s (stored in %s) is corrupt"),
+				    oid_to_hex(real), p->pack_name);
+		}
 		return -1;
 	}
 
@@ -1653,7 +1673,8 @@ int oid_object_info(struct repository *r,
 
 static void *read_object(struct repository *r,
 			 const struct object_id *oid, enum object_type *type,
-			 unsigned long *size)
+			 unsigned long *size,
+			 int die_if_corrupt)
 {
 	struct object_info oi = OBJECT_INFO_INIT;
 	void *content;
@@ -1661,7 +1682,8 @@ static void *read_object(struct repository *r,
 	oi.sizep = size;
 	oi.contentp = &content;
 
-	if (oid_object_info_extended(r, oid, &oi, 0) < 0)
+	if (oid_object_info_extended(r, oid, &oi, die_if_corrupt
+				     ? OBJECT_INFO_DIE_IF_CORRUPT : 0) < 0)
 		return NULL;
 	return content;
 }
@@ -1697,35 +1719,14 @@ void *read_object_file_extended(struct repository *r,
 				int lookup_replace)
 {
 	void *data;
-	const struct packed_git *p;
-	const char *path;
-	struct stat st;
 	const struct object_id *repl = lookup_replace ?
 		lookup_replace_object(r, oid) : oid;
 
 	errno = 0;
-	data = read_object(r, repl, type, size);
+	data = read_object(r, repl, type, size, 1);
 	if (data)
 		return data;
 
-	obj_read_lock();
-	if (errno && errno != ENOENT)
-		die_errno(_("failed to read object %s"), oid_to_hex(oid));
-
-	/* die if we replaced an object with one that does not exist */
-	if (repl != oid)
-		die(_("replacement %s not found for %s"),
-		    oid_to_hex(repl), oid_to_hex(oid));
-
-	if (!stat_loose_object(r, repl, &st, &path))
-		die(_("loose object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), path);
-
-	if ((p = has_packed_and_bad(r, repl)))
-		die(_("packed object %s (stored in %s) is corrupt"),
-		    oid_to_hex(repl), p->pack_name);
-	obj_read_unlock();
-
 	return NULL;
 }
 
@@ -2268,7 +2269,7 @@ int force_object_loose(const struct object_id *oid, time_t mtime)
 
 	if (has_loose_object(oid))
 		return 0;
-	buf = read_object(the_repository, oid, &type, &len);
+	buf = read_object(the_repository, oid, &type, &len, 0);
 	if (!buf)
 		return error(_("cannot read object for %s"), oid_to_hex(oid));
 	hdrlen = format_object_header(hdr, sizeof(hdr), type, len);
object-store.h +3 -0
index b1ec0bde82..98c1d67946 100644
@@ -445,6 +445,9 @@ struct object_info {
  */
 #define OBJECT_INFO_FOR_PREFETCH (OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)
 
+/* Die if object corruption (not just an object being missing) was detected. */
+#define OBJECT_INFO_DIE_IF_CORRUPT 32
+
 int oid_object_info_extended(struct repository *r,
 			     const struct object_id *,
 			     struct object_info *, unsigned flags);
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#82 lore raw

PATCH v6 4/4 commit: don't lazy-fetch commits

In reply to Jonathan Tan

When parsing commits, fail fast when the commit is missing or corrupt, instead of attempting to fetch them. This is done by inlining repo_read_object_file() and setting the flag that prevents fetching.

This is motivated by a situation in which through a bug (not necessarily through Git), there was corruption in the object store of a partial clone. In this particular case, the problem was exposed when "git gc" tried to expire reflogs, which calls repo_parse_commit(), which triggers fetches of the missing commits.

(There are other possible solutions to this problem including passing an argument from "git gc" to "git reflog" to inhibit all lazy fetches, but I think that this fix is at the wrong level - fixing "git reflog" means that this particular command works fine, or so we think (it will fail if it somehow needs to read a legitimately missing blob, say, a .gitmodules file), but fixing repo_parse_commit() will fix a whole class of bugs.)

---
 commit.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)
commit.c +13 -2
index 572301b80a..a02723f06b 100644
@@ -508,6 +508,17 @@ int repo_parse_commit_internal(struct repository *r,
 	enum object_type type;
 	void *buffer;
 	unsigned long size;
+	struct object_info oi = {
+		.typep = &type,
+		.sizep = &size,
+		.contentp = &buffer,
+	};
+	/*
+	 * Git does not support partial clones that exclude commits, so set
+	 * OBJECT_INFO_SKIP_FETCH_OBJECT to fail fast when an object is missing.
+	 */
+	int flags = OBJECT_INFO_LOOKUP_REPLACE | OBJECT_INFO_SKIP_FETCH_OBJECT |
+		OBJECT_INFO_DIE_IF_CORRUPT;
 	int ret;
 
 	if (!item)
@@ -516,8 +527,8 @@ int repo_parse_commit_internal(struct repository *r,
 		return 0;
 	if (use_commit_graph && parse_commit_in_graph(r, item))
 		return 0;
-	buffer = repo_read_object_file(r, &item->object.oid, &type, &size);
-	if (!buffer)
+
+	if (oid_object_info_extended(r, &item->object.oid, &oi, flags) < 0)
 		return quiet_on_missing ? -1 :
 			error("Could not read %s",
 			     oid_to_hex(&item->object.oid));
Signature · 1 line
2.39.0.314.g84b9a713c41-goog
#83 lore raw
In reply to Jonathan Tan
Quoted 7 lines · Jonathan Tan

Thanks everyone once again and sorry for the churn. Hopefully I got it right this time.

open_loose_object() is documented to return the path of the object we found, so I think we already have that covered (if we detect that an object is corrupt, it follows that we would already have found the object in the first place).

This version looks good to me. Thanks for your persistence. :) I think the end result is very nicely done.

-Peff

#84 lore raw
Quoted 5 lines · Jeff King
Quoted 3 lines · Jonathan Tan

Thanks everyone once again and sorry for the churn. Hopefully I got it right this time.

Quoted 7 lines
Quoted 4 lines

open_loose_object() is documented to return the path of the object we found, so I think we already have that covered (if we detect that an object is corrupt, it follows that we would already have found the object in the first place).

This version looks good to me. Thanks for your persistence. :) I think the end result is very nicely done.

Yeah, this looks good. Nothing added to or removed from the previous round other than what I found questionable during the review of that round.

Thanks, both.