Skip to contentSkip to conversation

receive-pack: use ODB transactions to stage object writes

PATCH Superseded Justin Tobler 3 participants 90 messages Jun 24 – Jul 13

lore
Share & tools
lore
#0 lore raw

cover receive-pack: use ODB transactions to stage object writes

Greetings,

This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.

Thanks,
-Justin

Justin Tobler (6):
object-file: rename files transaction prepare function
object-file: propagate files transaction errors odb/transaction: propagate begin errors odb/transaction: propagate commit errors odb/transaction: add transaction env interface builtin/receive-pack: stage incoming objects via ODB transactions

 builtin/add.c            |  2 +-
 builtin/receive-pack.c   | 46 ++++++++++--------------
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 77 +++++++++++++++++++++++++++++++---------
 object-file.h            |  7 ++--
 odb/source-files.c       |  9 ++---
 odb/source-inmemory.c    |  3 +-
 odb/source-loose.c       |  3 +-
 odb/source.h             |  9 +++--
 odb/transaction.c        | 38 +++++++++++++++-----
 odb/transaction.h        | 49 +++++++++++++++++++++----
 read-cache.c             |  2 +-
 14 files changed, 173 insertions(+), 78 deletions(-)

base-commit: ab776a62a78576513ee121424adb19597fbb7613

Signature · 1 line
2.54.0.105.g59ff4886a5
#1 lore raw

PATCH 1/6 object-file: rename files transaction prepare function

The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.

Rename the function to odb_transaction_files_prepare() accordingly.

---
 object-file.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
object-file.c +3 -3
index e3d92bbda2..a3eb8d71dd 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void prepare_loose_object_transaction(struct odb_transaction *base)
+static void odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf filename = STRBUF_INIT;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	odb_loose_path(loose, &filename, oid);
 
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,
 	int hdrlen;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
Signature · 1 line
2.54.0.105.g59ff4886a5
#2 lore raw

Re: [PATCH 1/6] object-file: rename files transaction prepare function

Quoted 6 lines · Justin Tobler

The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.

Rename the function to odb_transaction_files_prepare() accordingly.

Taken by itself this renaming does make sense, but there are many other function that follow the historical naming convention, like {fsync,flush}_loose_object_transaction(). Should we rename them for consistency with the new naming scheme, not necessarily as part of this series but with a todo comment to do so once the dust settles, or something?

#3 lore raw

Re: [PATCH 1/6] object-file: rename files transaction prepare function

Quoted 7 lines · Junio C Hamano
Quoted 5 lines · Justin Tobler

The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.

Quoted 8 lines
Quoted 1 line

Rename the function to odb_transaction_files_prepare() accordingly.

Taken by itself this renaming does make sense, but there are many other function that follow the historical naming convention, like {fsync,flush}_loose_object_transaction(). Should we rename them for consistency with the new naming scheme, not necessarily as part of this series but with a todo comment to do so once the dust settles, or something?

Ya, both {fsync,flush}_loose_object_transaction() are probably good candidates to be renamed to odb_transaction_files_{flush,flush} also. In the next version, I'll probably add another patch to do so accordingly.

-Justin

#4 lore raw

PATCH 2/6 object-file: propagate files transaction errors

In reply to Justin Tobler

The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle as needed.

---
 object-file.c      | 41 ++++++++++++++++++++++++++++-------------
 object-file.h      |  5 +++--
 odb/source-files.c |  6 +-----
 odb/transaction.h  |  2 +-
 4 files changed, 33 insertions(+), 21 deletions(-)
object-file.c +28 -13
index a3eb8d71dd..18c2df75fb 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;
+
+	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+
+	return 0;
 }
 
 static void fsync_loose_object_transaction(struct odb_transaction *base,
@@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 /*
  * Cleanup after batch-mode fsync_object_files.
  */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
+static int flush_loose_object_transaction(struct odb_transaction_files *transaction)
 {
 	struct strbuf temp_path = STRBUF_INIT;
 	struct tempfile *temp;
 
 	if (!transaction->objdir)
-		return;
+		return 0;
 
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
@@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
 	 * Make the object files visible in the primary ODB after their data is
 	 * fully durable.
 	 */
-	tmp_objdir_migrate(transaction->objdir);
+	if (tmp_objdir_migrate(transaction->objdir))
+		return -1;
+
 	transaction->objdir = NULL;
+
+	return 0;
 }
 
 /* Finalize a file on disk, and close it. */
@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (flush_loose_object_transaction(transaction))
+		return -1;
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 	struct object_database *odb = source->odb;
 
-	if (odb->transaction)
-		return NULL;
+	if (odb->transaction) {
+		*out = NULL;
+		return 0;
+	}
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
-	return &transaction->base;
+	return 0;
 }
object-file.h +3 -2
index 528c4e6e69..ac927fec07 100644
@@ -195,8 +195,9 @@ struct odb_transaction;
  * Tell the object database to optimize for adding
  * multiple objects. odb_transaction_files_commit must be called
  * to make new objects visible. If a transaction is already
- * pending, NULL is returned.
+ * pending, out is set to NULL.
  */
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out);
 
 #endif /* OBJECT_FILE_H */
odb/source-files.c +1 -5
index 5bdd042922..2545bd81d4 100644
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 static int odb_source_files_begin_transaction(struct odb_source *source,
 					      struct odb_transaction **out)
 {
-	struct odb_transaction *tx = odb_transaction_files_begin(source);
-	if (!tx)
-		return -1;
-	*out = tx;
-	return 0;
+	return odb_transaction_files_begin(source, out);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
odb/transaction.h +1 -1
index 854fda06f5..f4c1ebfaaa 100644
@@ -17,7 +17,7 @@ struct odb_transaction {
 	struct odb_source *source;
 
 	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	int (*commit)(struct odb_transaction *transaction);
 
 	/*
 	 * This callback is expected to write the given object stream into
Signature · 1 line
2.54.0.105.g59ff4886a5
#5 lore raw

Re: [PATCH 2/6] object-file: propagate files transaction errors

Quoted 5 lines · Justin Tobler

The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle as needed.

Missing a then? "to handle as needed" -> "to handle them as needed"

Makes sense. It always felt a bit off that those functions didn't have a way to signal errors to the caller.

object-file.c struct odb_transaction_files { · line 505

Quoted 13 lines
object-file.c +1 -1
index a3eb8d71dd..18c2df75fb 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);

By the way, is there any reason why those functions are still hosted in "object-file.c" instead of in "odb/source-files.c"? I should probably know, but I forgot.

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518

Quoted 12 lines
object-file.c +3 -3
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;

Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?

object-file.c static void fsync_loose_object_transaction(struct odb_transaction *base, · line 549

Quoted 6 lines
object-file.c +1 -1
@@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 /*
  * Cleanup after batch-mode fsync_object_files.
  */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
+static int flush_loose_object_transaction(struct odb_transaction_files *transaction)

Feels like this function should've been renamed in the preceding commit, as well.

object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac

Quoted 11 lines
object-file.c +1 -1
 {
 	struct strbuf temp_path = STRBUF_INIT;
 	struct tempfile *temp;
 
 	if (!transaction->objdir)
-		return;
+		return 0;
 
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
@@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac

There is a call to xmks_tempfile() hidden that can fail, but that failure is already handled in that function itself by dying.

object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac · line 578

Quoted 6 lines
object-file.c +2 -1
 	 * Make the object files visible in the primary ODB after their data is
 	 * fully durable.
 	 */
-	tmp_objdir_migrate(transaction->objdir);
+	if (tmp_objdir_migrate(transaction->objdir))
+		return -1;

Feels like another case of YOLOing it. The migration could have failed, but we just ignored that failure and never told the user about it. The result may be silent corruption, I assume?

Quoted 31 lines
object-file.c +11 -5
@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (flush_loose_object_transaction(transaction))
+		return -1;
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 	struct object_database *odb = source->odb;
 
-	if (odb->transaction)
-		return NULL;
+	if (odb->transaction) {
+		*out = NULL;
+		return 0;
+	}

object-file.c int read_loose_object(struct repository *repo, · line 1711

Quoted 10 lines
object-file.c +2 -1
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
-	return &transaction->base;
+	return 0;
 }

It's still somewhat fishy that we have this ODB-level transaction, but that's a preexisting issue and thus outside the scope of this patch series. Ideally though, it would be possible for there to be multiple transactions, and it would be the caller's responsibility for juggling these transactions. Just as it happens with reference transactions.

odb/transaction.h struct odb_transaction { · line 20

Quoted 10 lines
odb/transaction.h +1 -1
index 854fda06f5..f4c1ebfaaa 100644
@@ -17,7 +17,7 @@ struct odb_transaction {
 	struct odb_source *source;
 
 	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	int (*commit)(struct odb_transaction *transaction);

We might want to document the returned error code here.

Patrick

#6 lore raw

Re: [PATCH 2/6] object-file: propagate files transaction errors

Quoted 8 lines · Patrick Steinhardt
Quoted 5 lines · Justin Tobler

The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle as needed.

Missing a then? "to handle as needed" -> "to handle them as needed"

Will fix.

[snip]

Quoted 8 lines
Quoted 8 lines
object-file.c +0 -0
index a3eb8d71dd..18c2df75fb 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
Quoted 9 lines

object-file.c struct odb_transaction_files { · line 505

Quoted 5 lines
object-file.c +1 -1
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);

By the way, is there any reason why those functions are still hosted in "object-file.c" instead of in "odb/source-files.c"? I should probably know, but I forgot.

There are currently a couple spots in the "files" object write path in "object-file.c" that still reach into some of these transaction function that are not part of the generic ODB transaction interface. I'm hoping in a future followup series to detangle this a bit and be able to get all the "files" ODB transaction related code moved into "odb/source-files.c".

Quoted 7 lines
Quoted 7 lines
object-file.c +1 -1
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
Quoted 9 lines

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518

Quoted 5 lines
object-file.c +2 -2
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;

Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?

Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.

I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.

Quoted 9 lines

object-file.c static void fsync_loose_object_transaction(struct odb_transaction *base, · line 549

Quoted 6 lines
object-file.c +1 -1
@@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 /*
  * Cleanup after batch-mode fsync_object_files.
  */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
+static int flush_loose_object_transaction(struct odb_transaction_files *transaction)

Feels like this function should've been renamed in the preceding commit, as well.

Ya I agree, will address in the next version.

Quoted 15 lines

object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac

Quoted 11 lines
object-file.c +1 -1
 {
 	struct strbuf temp_path = STRBUF_INIT;
 	struct tempfile *temp;
 
 	if (!transaction->objdir)
-		return;
+		return 0;
 
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
@@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac

There is a call to xmks_tempfile() hidden that can fail, but that failure is already handled in that function itself by dying.

Quoted 10 lines

object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac · line 578

Quoted 6 lines
object-file.c +2 -1
 	 * Make the object files visible in the primary ODB after their data is
 	 * fully durable.
 	 */
-	tmp_objdir_migrate(transaction->objdir);
+	if (tmp_objdir_migrate(transaction->objdir))
+		return -1;

Feels like another case of YOLOing it. The migration could have failed, but we just ignored that failure and never told the user about it. The result may be silent corruption, I assume?

Ya in this case, if the migration failed, we would just continue on which would likely result in silent corruption.

Quoted 39 lines
Quoted 39 lines
object-file.c +12 -5
@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (flush_loose_object_transaction(transaction))
+		return -1;
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 	struct object_database *odb = source->odb;
 
-	if (odb->transaction)
-		return NULL;
+	if (odb->transaction) {
+		*out = NULL;
+		return 0;
+	}
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
Quoted 9 lines

object-file.c int read_loose_object(struct repository *repo, · line 1711

Quoted 3 lines
object-file.c +1 -1
-	return &transaction->base;
+	return 0;
 }

It's still somewhat fishy that we have this ODB-level transaction, but that's a preexisting issue and thus outside the scope of this patch series. Ideally though, it would be possible for there to be multiple transactions, and it would be the caller's responsibility for juggling these transactions. Just as it happens with reference transactions.

I completely agree. One of the current problems preventing this is that only a single instance of tmp_objdir is allowed and stored globally. This is done to keep atexit() cleanup simple.

My plan is to eventually convert all existing tmp_objdir callsites to use ODB transactions which should hopefully allow us to remove the need for a separate tmp_objdir system. At that point, we can also work to change how temp dir cleanup is handled at exit.

Another problem is that there are also a couple of ODB transaction callsites that require to know if there is already a pending transaction for the repository and the transaction has not been wired down to these callsites. My hope is that this can be addressed though as we expand ODB transaction usage for object writes.

Quoted 7 lines
Quoted 7 lines
odb/transaction.h +0 -0
index 854fda06f5..f4c1ebfaaa 100644
@@ -17,7 +17,7 @@ struct odb_transaction {
 	struct odb_source *source;
 
Quoted 5 lines

odb/transaction.h struct odb_transaction { · line 20

Quoted 3 lines
odb/transaction.h +1 -1
 	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	int (*commit)(struct odb_transaction *transaction);

We might want to document the returned error code here.

Will do in the next version.

Thanks,
-Justin

#7 lore raw

Re: [PATCH 2/6] object-file: propagate files transaction errors

Quoted 26 lines · Justin Tobler
Quoted 8 lines · Patrick Steinhardt
Quoted 7 lines · Justin Tobler
object-file.c +1 -1
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
Quoted 9 lines

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518

Quoted 5 lines
object-file.c +2 -2
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;

Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?

Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.

Quoted 3 lines

I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.

I think I got a bit ahead of myself. The existing callers of odb_transaction_files_prepare() still continue to ignore this error. So the behavior already does remain the same here.

-Justin

#8 lore raw

Re: [PATCH 2/6] object-file: propagate files transaction errors

Quoted 27 lines · Justin Tobler
Quoted 26 lines · Justin Tobler
Quoted 8 lines · Patrick Steinhardt
Quoted 7 lines · Justin Tobler
object-file.c +1 -1
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
Quoted 9 lines

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518

Quoted 5 lines
object-file.c +2 -2
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;

Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?

Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.

Quoted 7 lines
Quoted 3 lines

I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.

I think I got a bit ahead of myself. The existing callers of odb_transaction_files_prepare() still continue to ignore this error. So the behavior already does remain the same here.

Oh, well, okay. I think this behaviour is plain bad -- if the caller wants to have a transaction, then we should bail in case we cannot create one. But this doesn't need to be fixed in this patch series.

Patrick

#9 lore raw

Re: [PATCH 2/6] object-file: propagate files transaction errors

Quoted 32 lines · Patrick Steinhardt
Quoted 31 lines · Justin Tobler
Quoted 29 lines · Justin Tobler
Quoted 8 lines · Patrick Steinhardt
Quoted 7 lines · Justin Tobler
object-file.c +1 -1
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
Quoted 9 lines

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518

Quoted 5 lines
object-file.c +2 -2
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;

Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?

Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.

I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.

Quoted 7 lines
Quoted 3 lines

I think I got a bit ahead of myself. The existing callers of odb_transaction_files_prepare() still continue to ignore this error. So the behavior already does remain the same here.

Oh, well, okay. I think this behaviour is plain bad -- if the caller wants to have a transaction, then we should bail in case we cannot create one. But this doesn't need to be fixed in this patch series.

Ya, I tend to agree. The problem here is that odb_transaction_files_prepare() is being invoked lazily during the object write. This would be another argument against lazily creating the temporary directory though. In a followup series I'll explore removing this and investigate if it has any meaninful performance implications.

-Justin

#10 lore raw

Re: [PATCH 2/6] object-file: propagate files transaction errors

In reply to Justin Tobler
Quoted 20 lines · Justin Tobler
Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · Justin Tobler
object-file.c +0 -0
index a3eb8d71dd..18c2df75fb 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
Quoted 9 lines

object-file.c struct odb_transaction_files { · line 505

Quoted 5 lines
object-file.c +1 -1
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);

By the way, is there any reason why those functions are still hosted in "object-file.c" instead of in "odb/source-files.c"? I should probably know, but I forgot.

Quoted 6 lines

There are currently a couple spots in the "files" object write path in "object-file.c" that still reach into some of these transaction function that are not part of the generic ODB transaction interface. I'm hoping in a future followup series to detangle this a bit and be able to get all the "files" ODB transaction related code moved into "odb/source-files.c".

Makes sense. I also revisited that code a couple days ago, and the answer is "it's messy right now". Hopefully this will become easier to detangle as we make progress on pluggifying transactions.

Quoted 24 lines
Quoted 7 lines
Quoted 7 lines
object-file.c +1 -1
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
Quoted 9 lines

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518

Quoted 5 lines
object-file.c +2 -2
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;

Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?

Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.

Quoted 3 lines

I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.

Honestly I'd say that the change is a good one. I cannot think of a single reason to just blindly not create the transaction and proceed. But it certainly is something that should be documented as part of the commit message.

Quoted 48 lines
Quoted 38 lines
Quoted 38 lines
object-file.c +12 -5
@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (flush_loose_object_transaction(transaction))
+		return -1;
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 	struct object_database *odb = source->odb;
 
-	if (odb->transaction)
-		return NULL;
+	if (odb->transaction) {
+		*out = NULL;
+		return 0;
+	}
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
Quoted 9 lines

object-file.c int read_loose_object(struct repository *repo, · line 1711

Quoted 3 lines
object-file.c +1 -1
-	return &transaction->base;
+	return 0;
 }

It's still somewhat fishy that we have this ODB-level transaction, but that's a preexisting issue and thus outside the scope of this patch series. Ideally though, it would be possible for there to be multiple transactions, and it would be the caller's responsibility for juggling these transactions. Just as it happens with reference transactions.

Quoted 8 lines

I completely agree. One of the current problems preventing this is that only a single instance of tmp_objdir is allowed and stored globally. This is done to keep atexit() cleanup simple.

My plan is to eventually convert all existing tmp_objdir callsites to use ODB transactions which should hopefully allow us to remove the need for a separate tmp_objdir system. At that point, we can also work to change how temp dir cleanup is handled at exit.

Great.

Quoted 5 lines

Another problem is that there are also a couple of ODB transaction callsites that require to know if there is already a pending transaction for the repository and the transaction has not been wired down to these callsites. My hope is that this can be addressed though as we expand ODB transaction usage for object writes.

Yeah, agreed. Making the use of transactions explicit feels sensible to me.

Patrick

#11 lore raw

Re: [PATCH 2/6] object-file: propagate files transaction errors

In reply to Justin Tobler
Quoted 5 lines · Justin Tobler

The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle as needed.

"handle them as needed", perhaps.

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 516

Quoted 13 lines
object-file.c +2 -2
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");

If this fails, NULL is returned, and ...

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518

Quoted 4 lines
object-file.c +2 -2
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;

... we return -1 from here to signal an error now.

But callers of this function in write_loose_object(), and odb_source_loose_write_stream() are not prepared to react to such an error.

I guess this is nothing new. The callers ignored such an error from here in the original and proceeded writing the primary ODB anyway, and we continue to do so after this step.

Quoted 18 lines
object-file.c +2 -2
@@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 /*
  * Cleanup after batch-mode fsync_object_files.
  */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
+static int flush_loose_object_transaction(struct odb_transaction_files *transaction)
 {
 	struct strbuf temp_path = STRBUF_INIT;
 	struct tempfile *temp;
 
 	if (!transaction->objdir)
-		return;
+		return 0;
 
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
@@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
 	 * Make the object files visible in the primary ODB after their data is

object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac · line 583

Quoted 10 lines
object-file.c +5 -1
 	 * fully durable.
 	 */
-	tmp_objdir_migrate(transaction->objdir);
+	if (tmp_objdir_migrate(transaction->objdir))
+		return -1;
+
 	transaction->objdir = NULL;
+
+	return 0;
 }

The caller of this function does react to a failure of it, ...

Quoted 7 lines
object-file.c +1 -1
@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {

object-file.c int read_loose_object(struct repository *repo, · line 1691

Quoted 10 lines
object-file.c +4 -1
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (flush_loose_object_transaction(transaction))
+		return -1;
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }

... like this, which is good. Do we need an explicit "abort-transaction", or is that implicit?

Thanks.

#12 lore raw

Re: [PATCH 2/6] object-file: propagate files transaction errors

Quoted 9 lines · Junio C Hamano
Quoted 5 lines · Justin Tobler

The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle as needed.

"handle them as needed", perhaps.

Will fix, thanks

[snip]

Quoted 12 lines

The caller of this function does react to a failure of it, ...

Quoted 10 lines
object-file.c +1 -1
@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
Quoted 10 lines

object-file.c int read_loose_object(struct repository *repo, · line 1691

Quoted 7 lines
object-file.c +4 -1
-	flush_loose_object_transaction(transaction);
+	if (flush_loose_object_transaction(transaction))
+		return -1;
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }

... like this, which is good. Do we need an explicit "abort-transaction", or is that implicit?

So this is currently handled implicitly via tmp-objdir.c:remove_tmp_objdir() which gets registered as an atexit() handler. As long as the tmp_objdir global remains set, it will automatically get cleaned up.

In a subsequent series, I do plan to add odb_transaction_abort() to the transaction interface. It may make sense to also use that here to make the cleanup a bit more explicit though.

-Justin

#13 lore raw

PATCH 3/6 odb/transaction: propagate begin errors

In reply to Justin Tobler

When odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.

---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            |  3 ++-
 odb/transaction.c        | 16 +++++++++++-----
 odb/transaction.h        | 19 +++++++++++++++----
 read-cache.c             |  2 +-
 8 files changed, 33 insertions(+), 15 deletions(-)
builtin/add.c +1 -1
index c859f66519..3d5d9cfdb9 100644
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
builtin/unpack-objects.c +1 -1
index f3849bb654..d0136cdd99 100644
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
builtin/update-index.c +1 -1
index 3d6646c318..17f3ea284c 100644
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
index 184f7e2635..1a7dfed9cf 100644
@@ -490,7 +490,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	odb_transaction_commit(transaction);
object-file.c +2 -1
index 18c2df75fb..696f05dc2d 100644
@@ -1389,8 +1389,9 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 
 		if (flags & INDEX_WRITE_OBJECT) {
 			struct object_database *odb = the_repository->objects;
-			struct odb_transaction *transaction = odb_transaction_begin(odb);
+			struct odb_transaction *transaction;
 
+			odb_transaction_begin_or_die(odb, &transaction);
 			ret = odb_transaction_write_object_stream(odb->transaction,
 								  &stream,
 								  xsize_t(st->st_size),
odb/transaction.c +11 -5
index b16e07aebf..d3de01db50 100644
@@ -2,14 +2,20 @@
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)
 {
-	if (odb->transaction)
-		return NULL;
+	int ret;
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	if (odb->transaction) {
+		*out = NULL;
+		return 0;
+	}
 
-	return odb->transaction;
+	ret = odb_source_begin_transaction(odb->sources, out);
+	odb->transaction = *out;
+
+	return ret;
 }
 
 void odb_transaction_commit(struct odb_transaction *transaction)
odb/transaction.h +15 -4
index f4c1ebfaaa..cd6d50f2e5 100644
@@ -1,6 +1,8 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "git-compat-util.h"
+#include "gettext.h"
 #include "odb.h"
 #include "odb/source.h"
 
@@ -33,11 +35,20 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. If the ODB already has a pending transaction, `out` is set to NULL.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}
 
 /*
  * Commits an ODB transaction making the written objects visible. If the
read-cache.c +1 -1
index 21ca58beea..db0bfa60fe 100644
@@ -4042,7 +4042,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	odb_transaction_commit(transaction);
 
Signature · 1 line
2.54.0.105.g59ff4886a5
#14 lore raw

Re: [PATCH 3/6] odb/transaction: propagate begin errors

Quoted 11 lines · Justin Tobler
odb/transaction.c +2 -1
index b16e07aebf..d3de01db50 100644
@@ -2,14 +2,20 @@
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)

odb/transaction.c line 13

Quoted 10 lines
odb/transaction.c +5 -3
 {
-	if (odb->transaction)
-		return NULL;
+	int ret;
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	if (odb->transaction) {
+		*out = NULL;
+		return 0;
+	}

Hm. So we may return successful, but not set the out pointer to a transaction. And...

Quoted 16 lines
odb/transaction.h +4 -3
index f4c1ebfaaa..cd6d50f2e5 100644
@@ -33,11 +35,20 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. If the ODB already has a pending transaction, `out` is set to NULL.
  */

odb/transaction.h struct odb_transaction { · line 51

Quoted 10 lines
odb/transaction.h +9 -1
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}

... we don't special-case that here, either. So a caller may invoke the function, not die, but it might still not have a valid transaction. That feels wrong to me.

Patrick

#15 lore raw

Re: [PATCH 3/6] odb/transaction: propagate begin errors

Quoted 48 lines · Patrick Steinhardt
Quoted 11 lines · Justin Tobler
odb/transaction.c +2 -1
index b16e07aebf..d3de01db50 100644
@@ -2,14 +2,20 @@
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)

odb/transaction.c line 13

Quoted 10 lines
odb/transaction.c +5 -3
 {
-	if (odb->transaction)
-		return NULL;
+	int ret;
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	if (odb->transaction) {
+		*out = NULL;
+		return 0;
+	}

Hm. So we may return successful, but not set the out pointer to a transaction. And...

Quoted 22 lines
odb/transaction.h +9 -4
index f4c1ebfaaa..cd6d50f2e5 100644
@@ -33,11 +35,20 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. If the ODB already has a pending transaction, `out` is set to NULL.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
Quoted 8 lines

odb/transaction.h struct odb_transaction { · line 51

Quoted 4 lines
odb/transaction.h +4 -0
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}

... we don't special-case that here, either. So a caller may invoke the function, not die, but it might still not have a valid transaction. That feels wrong to me.

Ya, the original idea for a NULL transaction to signal that there is already a transaction in flight and nested ODB transaction users to use that to determine whether they needed to start a new transaction or not. I completely agree thought that this is rather ackward.

Instead we could just make the callers that are worried about potentially nested transactions handle this explicitly. I'll do that in the next version.

-Justin

#16 lore raw

PATCH 4/6 odb/transaction: propagate commit errors

In reply to Justin Tobler

When odb_transaction_commit() is invoked, the return value of the backend commit callback is silently discarded. A backend has no way to signal that committing failed, such as when the "files" backend cannot migrate its temporary object directory into the permanent ODB.

In a subsequent commit, git-receive-pack(1) starts using ODB transaction to stage objects and consequently cares about such failures so it can handle the error appropriately. Change the commit callback signature to return an int error code and have odb_transaction_commit() forward it accordingly.

---
 odb/transaction.c | 13 ++++++++++---
 odb/transaction.h |  2 +-
 2 files changed, 11 insertions(+), 4 deletions(-)
odb/transaction.c +10 -3
index d3de01db50..b20d6a16f8 100644
@@ -18,19 +18,26 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
+	if (ret)
+		return ret;
+
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return 0;
 }
 
 int odb_transaction_write_object_stream(struct odb_transaction *transaction,
odb/transaction.h +1 -1
index cd6d50f2e5..7898770071 100644
@@ -54,7 +54,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
  * Commits an ODB transaction making the written objects visible. If the
  * specified transaction is NULL, the function is a no-op.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);
 
 /*
  * Writes the object in the provided stream into the transaction. The resulting
Signature · 1 line
2.54.0.105.g59ff4886a5
#17 lore raw

Re: [PATCH 4/6] odb/transaction: propagate commit errors

odb/transaction.h static inline void odb_transaction_begin_or_die(struct object_database *odb, · line 57

Quoted 10 lines · Justin Tobler
odb/transaction.h +1 -1
index cd6d50f2e5..7898770071 100644
@@ -54,7 +54,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
  * Commits an ODB transaction making the written objects visible. If the
  * specified transaction is NULL, the function is a no-op.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);

Should the function comment be amended, as well? We should definitely point out that calling this with a NULL transaction also returns success.

Patrick

#18 lore raw

Re: [PATCH 4/6] odb/transaction: propagate commit errors

Quoted 15 lines · Patrick Steinhardt

odb/transaction.h static inline void odb_transaction_begin_or_die(struct object_database *odb, · line 57

Quoted 10 lines · Justin Tobler
odb/transaction.h +1 -1
index cd6d50f2e5..7898770071 100644
@@ -54,7 +54,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
  * Commits an ODB transaction making the written objects visible. If the
  * specified transaction is NULL, the function is a no-op.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);

Should the function comment be amended, as well? We should definitely point out that calling this with a NULL transaction also returns success.

Will do in the next version.

Thanks,
-Justin

#19 lore raw

PATCH 5/6 odb/transaction: add transaction env interface

In reply to Justin Tobler

The ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need to access to uncommitted objects or write new objects in the staging area though.

Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction staging area.

---
 object-file.c     | 11 +++++++++++
 odb/transaction.c |  8 ++++++++
 odb/transaction.h | 19 +++++++++++++++++++
 3 files changed, 38 insertions(+)
object-file.c +11 -0
index 696f05dc2d..14064d188a 100644
@@ -1691,6 +1691,16 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	return 0;
 }
 
+static const char **odb_transaction_files_env(struct odb_transaction *base)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+
+	odb_transaction_files_prepare(&transaction->base);
+
+	return tmp_objdir_env(transaction->objdir);
+}
+
 int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out)
 {
@@ -1706,6 +1716,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	transaction->base.env = odb_transaction_files_env;
 	*out = &transaction->base;
 
 	return 0;
odb/transaction.c +8 -0
index b20d6a16f8..20d3f43f54 100644
@@ -46,3 +46,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 {
 	return transaction->write_object_stream(transaction, stream, len, oid);
 }
+
+const char **odb_transaction_env(struct odb_transaction *transaction)
+{
+	if (!transaction)
+		return NULL;
+
+	return transaction->env(transaction);
+}
odb/transaction.h +19 -0
index 7898770071..536458297b 100644
@@ -32,6 +32,16 @@ struct odb_transaction {
 	int (*write_object_stream)(struct odb_transaction *transaction,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to return a NULL-terminated array of
+	 * environment variables that a child process should inherit so
+	 * that its object writes participate in the transaction. The
+	 * returned array is owned by the backend and remains valid until
+	 * the transaction ends. May return NULL when the backend does not
+	 * need to expose any state to child processes.
+	 */
+	const char **(*env)(struct odb_transaction *transaction);
 };
 
 /*
@@ -65,4 +75,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 					struct odb_write_stream *stream,
 					size_t len, struct object_id *oid);
 
+/*
+ * Returns a NULL-terminated array of environment variables that a child
+ * process should inherit so that its object writes participate in the
+ * transaction, suitable for passing via child_process.env. Returns NULL if
+ * the transaction is NULL or the backend does not expose any state to child
+ * processes.
+ */
+const char **odb_transaction_env(struct odb_transaction *transaction);
+
 #endif
Signature · 1 line
2.54.0.105.g59ff4886a5
#20 lore raw

Re: [PATCH 5/6] odb/transaction: add transaction env interface

Quoted 3 lines · Justin Tobler

The ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need to access to uncommitted objects or write new objects in the

s/may need to access to/may need access to/

Quoted 5 lines

staging area though.

Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction staging area.

Possessive s is missing, I think.

Quoted 11 lines
object-file.c +3 -0
index 696f05dc2d..14064d188a 100644
@@ -1691,6 +1691,16 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	return 0;
 }
 
+static const char **odb_transaction_files_env(struct odb_transaction *base)
+{
+	struct odb_transaction_files *transaction =

object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1706

Quoted 10 lines
object-file.c +7 -0
+		container_of(base, struct odb_transaction_files, base);
+
+	odb_transaction_files_prepare(&transaction->base);
+
+	return tmp_objdir_env(transaction->objdir);
+}
+
 int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out)
 {

Makes sense. Transactions may have a different way to quarantine the write than using a quarantine directory. So making this functionality pluggable so that backends may expose a separate set of environment variables feels sensible.

Quoted 8 lines
odb/transaction.h +0 -0
index 7898770071..536458297b 100644
@@ -32,6 +32,16 @@ struct odb_transaction {
 	int (*write_object_stream)(struct odb_transaction *transaction,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);

odb/transaction.h struct odb_transaction { · line 44

Quoted 10 lines
odb/transaction.h +10 -0
+
+	/*
+	 * This callback is expected to return a NULL-terminated array of
+	 * environment variables that a child process should inherit so
+	 * that its object writes participate in the transaction. The
+	 * returned array is owned by the backend and remains valid until
+	 * the transaction ends. May return NULL when the backend does not
+	 * need to expose any state to child processes.
+	 */
+	const char **(*env)(struct odb_transaction *transaction);

Would it make more sense to adapt this function so that:

- It receives a struct strvec as input that the environment variables are to be amended to.

- It returns a normal error code to indicate errors?

Patrick

#21 lore raw

Re: [PATCH 5/6] odb/transaction: add transaction env interface

Quoted 6 lines · Patrick Steinhardt
Quoted 3 lines · Justin Tobler

The ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need to access to uncommitted objects or write new objects in the

s/may need to access to/may need access to/

Will fix.

Quoted 7 lines
Quoted 5 lines

staging area though.

Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction staging area.

Possessive s is missing, I think.

Yes. :)

[snip]

Quoted 13 lines

odb/transaction.h struct odb_transaction { · line 44

Quoted 10 lines
odb/transaction.h +10 -0
+
+	/*
+	 * This callback is expected to return a NULL-terminated array of
+	 * environment variables that a child process should inherit so
+	 * that its object writes participate in the transaction. The
+	 * returned array is owned by the backend and remains valid until
+	 * the transaction ends. May return NULL when the backend does not
+	 * need to expose any state to child processes.
+	 */
+	const char **(*env)(struct odb_transaction *transaction);

Would it make more sense to adapt this function so that:

Quoted 4 lines

- It receives a struct strvec as input that the environment variables are to be amended to.

- It returns a normal error code to indicate errors?

Ya, that is probably a more sensible interface as it would be nice to be able to signal an error. Will do in the next version.

Thanks,
-Justin

#22 lore raw

PATCH 6/6 builtin/receive-pack: stage incoming objects via ODB transactions

In reply to Justin Tobler

Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.

Note that the temporary directory created for git-receive-pack(1) is eagerly created and uses a different prefix name. This behavior is special cased in the "files" backend by having odb_transaction_begin() callers that require this behavior provide an ODB_TRANSACTION_RECEIVE flag.

---
 builtin/add.c            |  2 +-
 builtin/receive-pack.c   | 46 ++++++++++++++++------------------------
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 22 ++++++++++++++++---
 object-file.h            |  4 +++-
 odb/source-files.c       |  5 +++--
 odb/source-inmemory.c    |  3 ++-
 odb/source-loose.c       |  3 ++-
 odb/source.h             |  9 +++++---
 odb/transaction.c        |  5 +++--
 odb/transaction.h        | 13 ++++++++----
 read-cache.c             |  2 +-
 14 files changed, 70 insertions(+), 50 deletions(-)
builtin/add.c +1 -1
index 3d5d9cfdb9..60ffbede2b 100644
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
builtin/receive-pack.c +18 -28

129 lines not on this page · raw message

builtin/unpack-objects.c +1 -1
index d0136cdd99..c3d0fc7507 100644
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
builtin/update-index.c +1 -1
index 17f3ea284c..bf6ea60ef4 100644
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
index 1a7dfed9cf..ed05acc4c7 100644
@@ -490,7 +490,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	odb_transaction_commit(transaction);
object-file.c +19 -3
index 14064d188a..e7958753ec 100644
@@ -497,6 +497,7 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
+	const char *prefix;
 };
 
 static int odb_transaction_files_prepare(struct odb_transaction *base)
@@ -513,7 +514,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)
 	if (!transaction || transaction->objdir)
 		return 0;
 
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
+	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
 	if (!transaction->objdir)
 		return -1;
 
@@ -1391,7 +1392,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 			struct object_database *odb = the_repository->objects;
 			struct odb_transaction *transaction;
 
-			odb_transaction_begin_or_die(odb, &transaction);
+			odb_transaction_begin_or_die(odb, &transaction, 0);
 			ret = odb_transaction_write_object_stream(odb->transaction,
 								  &stream,
 								  xsize_t(st->st_size),
@@ -1702,7 +1703,8 @@ static const char **odb_transaction_files_env(struct odb_transaction *base)
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 	struct object_database *odb = source->odb;
@@ -1717,6 +1719,20 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different prefix.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+
 	*out = &transaction->base;
 
 	return 0;
object-file.h +3 -1
index ac927fec07..fe098d54cb 100644
@@ -5,6 +5,7 @@
 #include "object.h"
 #include "odb.h"
 #include "odb/source-loose.h"
+#include "odb/transaction.h"
 
 /* The maximum size for an object header. */
 #define MAX_HEADER_LEN 32
@@ -198,6 +199,7 @@ struct odb_transaction;
  * pending, out is set to NULL.
  */
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out);
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags);
 
 #endif /* OBJECT_FILE_H */
odb/source-files.c +3 -2
index 2545bd81d4..534f48aad9 100644
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_files_begin_transaction(struct odb_source *source,
-					      struct odb_transaction **out)
+					      struct odb_transaction **out,
+					      enum odb_transaction_flags flags)
 {
-	return odb_transaction_files_begin(source, out);
+	return odb_transaction_files_begin(source, out, flags);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
odb/source-inmemory.c +2 -1
index e004566d76..9644d9d474 100644
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source,
 }
 
 static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED,
-						 struct odb_transaction **out UNUSED)
+						 struct odb_transaction **out UNUSED,
+						 enum odb_transaction_flags flags UNUSED)
 {
 	return error("in-memory source does not support transactions");
 }
odb/source-loose.c +2 -1
index 66e6bb8d3f..57c91986b4 100644
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED,
-					      struct odb_transaction **out UNUSED)
+					      struct odb_transaction **out UNUSED,
+					      enum odb_transaction_flags flags UNUSED)
 {
 	/* TODO: this is a known omission that we'll want to address eventually. */
 	return error("loose source does not support transactions");
odb/source.h +6 -3
index 2192a101b8..3790d03ff2 100644
@@ -3,6 +3,7 @@
 
 #include "object.h"
 #include "odb.h"
+#include "odb/transaction.h"
 
 enum odb_source_type {
 	/*
@@ -228,7 +229,8 @@ struct odb_source {
 	 * negative error code otherwise.
 	 */
 	int (*begin_transaction)(struct odb_source *source,
-				 struct odb_transaction **out);
+				 struct odb_transaction **out,
+				 enum odb_transaction_flags flags);
 
 	/*
 	 * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,
  * Returns 0 on success, a negative error code otherwise.
  */
 static inline int odb_source_begin_transaction(struct odb_source *source,
-					       struct odb_transaction **out)
+					       struct odb_transaction **out,
+					       enum odb_transaction_flags flags)
 {
-	return source->begin_transaction(source, out);
+	return source->begin_transaction(source, out, flags);
 }
 
 #endif
odb/transaction.c +3 -2
index 20d3f43f54..34c212020c 100644
@@ -3,7 +3,8 @@
 #include "odb/transaction.h"
 
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out)
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags)
 {
 	int ret;
 
@@ -12,7 +13,7 @@ int odb_transaction_begin(struct object_database *odb,
 		return 0;
 	}
 
-	ret = odb_source_begin_transaction(odb->sources, out);
+	ret = odb_source_begin_transaction(odb->sources, out, flags);
 	odb->transaction = *out;
 
 	return ret;
odb/transaction.h +9 -4
index 536458297b..78392ff13d 100644
@@ -4,7 +4,6 @@
 #include "git-compat-util.h"
 #include "gettext.h"
 #include "odb.h"
-#include "odb/source.h"
 
 /*
  * A transaction may be started for an object database prior to writing new
@@ -44,6 +43,10 @@ struct odb_transaction {
 	const char **(*env)(struct odb_transaction *transaction);
 };
 
+enum odb_transaction_flags {
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()
@@ -51,12 +54,14 @@ struct odb_transaction {
  * error. If the ODB already has a pending transaction, `out` is set to NULL.
  */
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out);
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags);
 
 static inline void odb_transaction_begin_or_die(struct object_database *odb,
-						struct odb_transaction **out)
+						struct odb_transaction **out,
+						enum odb_transaction_flags flags)
 {
-	if (odb_transaction_begin(odb, out))
+	if (odb_transaction_begin(odb, out, flags))
 		die(_("failed to start ODB transaction"));
 }
 
read-cache.c +1 -1
index db0bfa60fe..35bfb25576 100644
@@ -4042,7 +4042,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	odb_transaction_commit(transaction);
 
Signature · 1 line
2.54.0.105.g59ff4886a5
#23 lore raw

Re: [PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions

Quoted 10 lines · Justin Tobler

Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.

Note that the temporary directory created for git-receive-pack(1) is eagerly created and uses a different prefix name. This behavior is

A different prefix name compared to what?

Quoted 3 lines

special cased in the "files" backend by having odb_transaction_begin() callers that require this behavior provide an ODB_TRANSACTION_RECEIVE flag.

Okay. I guess this is to retain existing behaviour where the temporary directory is created lazily everywhere else. Makes me wonder whether we should eventually change this to just unconditionally create the directory in all cases so that we can drop this new flag.

It might've also made sense to split this commit up into two: one to introduce the flag parameter, and then one to do the changes to git-receive-pack(1).

builtin/receive-pack.c static enum { · line 117

Quoted 13 lines
builtin/receive-pack.c +0 -2
index 19eb6a1b61..ee8e03e2ab 100644
@@ -112,8 +112,6 @@ static enum {
 } use_keepalive;
 static int keepalive_in_sec = 5;
 
-static struct tmp_objdir *tmp_objdir;
-
 static struct proc_receive_ref {
 	unsigned int want_add:1,
 		     want_delete:1,

I assume the goal is that we convert all other users of the tmp-objdir subsystem to also use transactions eventually, so that this becomes an implementation detail fo the files transaction?

builtin/receive-pack.c static void execute_commands(struct command *commands, · line 2116

Quoted 13 lines
builtin/receive-pack.c +1 -2
@@ -2106,14 +2104,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
+	if (odb_transaction_commit(the_repository->objects->transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;

We don't need to unset the transaction because that's what odb_transaction_commit() already does for us, I assume?

builtin/receive-pack.c static void push_header_arg(struct strvec *args, struct pack_header *hdr) · line 2330

Quoted 10 lines
builtin/receive-pack.c +2 -1
@@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;

It feels a bit weird that we sometimes pass the transaction as parameter, whereas othertimes we access it via the_repository.

Quoted 9 lines
builtin/receive-pack.c +0 -5
@@ -2351,20 +2349,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		strvec_push(&child.args, alt_shallow_file);
 	}
 
-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-	if (!tmp_objdir) {
-		if (err_fd > 0)
-			close(err_fd);
-		return "unable to create temporary object directory";

builtin/receive-pack.c static const char *unpack(int err_fd, struct shallow_info *si) · line 2352

Quoted 10 lines
builtin/receive-pack.c +1 -9
-	}
-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
-
-	/*
-	 * Normally we just pass the tmp_objdir environment to the child
-	 * processes that do the heavy lifting, but we may need to see these
-	 * objects ourselves to set up shallow information.
-	 */
-	tmp_objdir_add_as_alternate(tmp_objdir);
+	strvec_pushv(&child.env, odb_transaction_env(transaction));

Interesting, this here seems like a change in behaviour. Previously we added the transactions as an alternate, but now we only propagate it via the environment. I didn't see this mentioned in the commit message.

builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2698

Quoted 7 lines
builtin/receive-pack.c +2 -1
@@ -2707,7 +2694,10 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
+				unpack_status = "unable to start ODB transaction";

s/ODB/object/

This may be visible to the user, and "ODB" may mean nothing to them.

Quoted 22 lines
object-file.c +6 -1
index 14064d188a..e7958753ec 100644
@@ -1702,7 +1703,8 @@ static const char **odb_transaction_files_env(struct odb_transaction *base)
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 	struct object_database *odb = source->odb;
@@ -1717,6 +1719,20 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*

object-file.c int odb_transaction_files_begin(struct odb_source *source, · line 1735

Quoted 10 lines
object-file.c +10 -0
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different prefix.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+

Okay, makes sense. I really wonder whether we need to insist this much on the exact name used by this, but better be safe than sorry for now I guess.

And as mentioned before, I also wonder whether it really makes sense to have the lazy creation of the tmp-objdir. Maybe add a NEEDSWORK item here that mentions we want to investigate whether this is even needed at all?

odb/transaction.h struct odb_transaction { · line 48

Quoted 11 lines
odb/transaction.h +3 -0
index 536458297b..78392ff13d 100644
@@ -44,6 +43,10 @@ struct odb_transaction {
 	const char **(*env)(struct odb_transaction *transaction);
 };
 
+enum odb_transaction_flags {
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};

It's not clear at all what this flag does based on its name, so we should have documentation for it.

Patrick

#24 lore raw

Re: [PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions

Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · Justin Tobler

Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.

Quoted 4 lines
Quoted 2 lines

Note that the temporary directory created for git-receive-pack(1) is eagerly created and uses a different prefix name. This behavior is

A different prefix name compared to what?

Currently the temporary directories created for ODB transactions all use the prefix "bulk-fsync". The temp dir created by git-receive-pack(1) is expected to have the prefix "incoming".

Quoted 8 lines
Quoted 3 lines

special cased in the "files" backend by having odb_transaction_begin() callers that require this behavior provide an ODB_TRANSACTION_RECEIVE flag.

Okay. I guess this is to retain existing behaviour where the temporary directory is created lazily everywhere else. Makes me wonder whether we should eventually change this to just unconditionally create the directory in all cases so that we can drop this new flag.

It would be nice to not have to have a flag here, but if we want to also keep the existing temp dir prefixes, we would also need to keep the flags.

Quoted 3 lines

It might've also made sense to split this commit up into two: one to introduce the flag parameter, and then one to do the changes to git-receive-pack(1).

Ya, I think you are right. I actually originally had it this way but squashed these two commits together for some reason. I'll split them back up in the next version.

Quoted 8 lines
Quoted 8 lines
builtin/receive-pack.c +0 -0
index 19eb6a1b61..ee8e03e2ab 100644
@@ -112,8 +112,6 @@ static enum {
 } use_keepalive;
 static int keepalive_in_sec = 5;
 
Quoted 9 lines

builtin/receive-pack.c static enum { · line 117

Quoted 5 lines
builtin/receive-pack.c +0 -2
-static struct tmp_objdir *tmp_objdir;
-
 static struct proc_receive_ref {
 	unsigned int want_add:1,
 		     want_delete:1,

I assume the goal is that we convert all other users of the tmp-objdir subsystem to also use transactions eventually, so that this becomes an implementation detail fo the files transaction?

Yes, that is exactly my plan :)

Quoted 8 lines
Quoted 8 lines
builtin/receive-pack.c +1 -1
@@ -2106,14 +2104,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
+	if (odb_transaction_commit(the_repository->objects->transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
Quoted 8 lines

builtin/receive-pack.c static void execute_commands(struct command *commands, · line 2116

Quoted 5 lines
builtin/receive-pack.c +0 -1
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;

We don't need to unset the transaction because that's what odb_transaction_commit() already does for us, I assume?

That is correct. The tmp_objdir global is actually removed and now competely managed by the ODB transaction. We can probably actuall remove the "tmp-objdir" include now too. I'll do that in the next version.

Quoted 4 lines
Quoted 4 lines
builtin/receive-pack.c +0 -0
@@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
Quoted 9 lines

builtin/receive-pack.c static void push_header_arg(struct strvec *args, struct pack_header *hdr) · line 2330

Quoted 6 lines
builtin/receive-pack.c +2 -1
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;

It feels a bit weird that we sometimes pass the transaction as parameter, whereas othertimes we access it via the_repository.

That's fair. I was trying to avoid the churn of wiring to all its callsites, but it's probably best to be consistent. Maybe it would be fine to just create a transaction global like we do for the reference transaction?

Quoted 15 lines
Quoted 15 lines
builtin/receive-pack.c +0 -11
@@ -2351,20 +2349,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		strvec_push(&child.args, alt_shallow_file);
 	}
 
-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-	if (!tmp_objdir) {
-		if (err_fd > 0)
-			close(err_fd);
-		return "unable to create temporary object directory";
-	}
-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
-
-	/*
-	 * Normally we just pass the tmp_objdir environment to the child
-	 * processes that do the heavy lifting, but we may need to see these
Quoted 8 lines

builtin/receive-pack.c static const char *unpack(int err_fd, struct shallow_info *si) · line 2352

Quoted 4 lines
builtin/receive-pack.c +1 -3
-	 * objects ourselves to set up shallow information.
-	 */
-	tmp_objdir_add_as_alternate(tmp_objdir);
+	strvec_pushv(&child.env, odb_transaction_env(transaction));

Interesting, this here seems like a change in behaviour. Previously we added the transactions as an alternate, but now we only propagate it via the environment. I didn't see this mentioned in the commit message.

That's not quite correct. By using the ODB transaction interface, the underlying tmp_objdir is managed transparently. When we begin the ODB transaction, the temp directory that gets created is automatically created and applied as the primary ODB. This ultimately produces an equivalent result, its just now set up when the transaction is started (which happens a little bit earlier).

The only behavior change here is that the temp directory is being applied as the primary instead of an alternate in the main git-receive-pack(1) process. Since all writes though are handled by the child processes that get spawned, this shouldn't really matter though.

This is certainly rather confusing though, and should be mentioned in the commit message. I'll do this in the next version.

Quoted 11 lines

builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2698

Quoted 7 lines
builtin/receive-pack.c +2 -1
@@ -2707,7 +2694,10 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
+				unpack_status = "unable to start ODB transaction";

s/ODB/object/

This may be visible to the user, and "ODB" may mean nothing to them.

Will change in the next version.

Quoted 37 lines
Quoted 22 lines
object-file.c +6 -1
index 14064d188a..e7958753ec 100644
@@ -1702,7 +1703,8 @@ static const char **odb_transaction_files_env(struct odb_transaction *base)
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 	struct object_database *odb = source->odb;
@@ -1717,6 +1719,20 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*

object-file.c int odb_transaction_files_begin(struct odb_source *source, · line 1735

Quoted 10 lines
object-file.c +10 -0
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different prefix.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+

Okay, makes sense. I really wonder whether we need to insist this much on the exact name used by this, but better be safe than sorry for now I guess.

Quoted 4 lines

And as mentioned before, I also wonder whether it really makes sense to have the lazy creation of the tmp-objdir. Maybe add a NEEDSWORK item here that mentions we want to investigate whether this is even needed at all?

Ya, I was worried that there would be some reason the temp dir prefix should remain the same or there would be some performance reason to lazily create directories. I didn't investigate too deeply though. I'll add a NEEDSWORK comment to make note of this.

Quoted 8 lines
Quoted 8 lines
odb/transaction.h +0 -0
index 536458297b..78392ff13d 100644
@@ -44,6 +43,10 @@ struct odb_transaction {
 	const char **(*env)(struct odb_transaction *transaction);
 };
 
Quoted 6 lines

odb/transaction.h struct odb_transaction { · line 48

Quoted 3 lines
odb/transaction.h +3 -0
+enum odb_transaction_flags {
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};

It's not clear at all what this flag does based on its name, so we should have documentation for it.

Will update.

Thanks,
-Justin

#25 lore raw

Re: [PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions

Quoted 23 lines · Justin Tobler
Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · Justin Tobler

Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.

Quoted 4 lines
Quoted 2 lines

Note that the temporary directory created for git-receive-pack(1) is eagerly created and uses a different prefix name. This behavior is

A different prefix name compared to what?

Currently the temporary directories created for ODB transactions all use the prefix "bulk-fsync". The temp dir created by git-receive-pack(1) is expected to have the prefix "incoming".

Quoted 4 lines
Quoted 3 lines

special cased in the "files" backend by having odb_transaction_begin() callers that require this behavior provide an ODB_TRANSACTION_RECEIVE flag.

Quoted 8 lines
Quoted 4 lines

Okay. I guess this is to retain existing behaviour where the temporary directory is created lazily everywhere else. Makes me wonder whether we should eventually change this to just unconditionally create the directory in all cases so that we can drop this new flag.

It would be nice to not have to have a flag here, but if we want to also keep the existing temp dir prefixes, we would also need to keep the flags.

Fair. Makes me wonder whether we really need to keep the exact same naming for this temporary directory. This is so deep into internals that I'm not sure whether we really need to treat this as part of our interface. I'm rather inclined to say it's not necessary.

In any case, I think it's fine to defer that discussion and keep this as-is for now. But we might keep it in the back of our minds and maybe simplify this in a subsequent patch series.

Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
builtin/receive-pack.c +0 -0
index 19eb6a1b61..ee8e03e2ab 100644
@@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
Quoted 9 lines

builtin/receive-pack.c static void push_header_arg(struct strvec *args, struct pack_header *hdr) · line 2330

Quoted 6 lines
builtin/receive-pack.c +2 -1
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;

It feels a bit weird that we sometimes pass the transaction as parameter, whereas othertimes we access it via the_repository.

Quoted 4 lines

That's fair. I was trying to avoid the churn of wiring to all its callsites, but it's probably best to be consistent. Maybe it would be fine to just create a transaction global like we do for the reference transaction?

My first kneejerk reaction was "no", but then I noticed that the global variable you're talking about is local to "builtin/receive-pack.c". So that might be an okayish solution.

Thanks!

Patrick

#26 lore raw
In reply to Justin Tobler
Quoted 10 lines · Justin Tobler

Greetings,

This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.

Thanks, this was a pleasant read. I've got a bunch of comments, but overall I really like the direction of this patch series.

Patrick

#27 lore raw
In reply to Justin Tobler
Quoted 11 lines · Justin Tobler

Greetings,

This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.

Quoted 2 lines

Thanks,
-Justin

The integration cycle this morning was somehow more painful than other cycles.

FYI, this topic had some interactions with ps/odb-source-packed and ps/refs-avoid-chdir-notify-reparent and needed the following evil-merge fix to make it build.

commit 721c15c1d5ef8f8aab8b9f50463e979e890b1ab6

Author: Junio C Hamano <[email protected]>
Date:   Wed Jun 24 12:45:35 2026 -0700

merge-fix/jt/receive-pack-use-odb-transactions

conflict with ps/odb-source-packed and ps/refs-avoid-chdir-notify-reparent

odb/source-packed.c +2 -1
index c3c26fb53e..b9231a8da0 100644
@@ -545,7 +545,8 @@ static int odb_source_packed_write_object_stream(struct odb_source *source UNUSE
 }
 
 static int odb_source_packed_begin_transaction(struct odb_source *source UNUSED,
-					       struct odb_transaction **out UNUSED)
+					       struct odb_transaction **out UNUSED,
+					       enum odb_transaction_flags flags UNUSED)
 {
 	return error("packed backend cannot begin transactions");
 }
t/unit-tests/u-reftable-table.c +4 -2
index a35bbc8004..dd5db06534 100644
@@ -235,7 +235,8 @@ void test_reftable_table__seek_invalid_log_offset(void)
 	uint8_t *footer;
 
 	cl_reftable_write_to_buf(&buf, refs, ARRAY_SIZE(refs),
-				 logs, ARRAY_SIZE(logs), NULL);
+				 logs, ARRAY_SIZE(logs),
+				 REFTABLE_HASH_SHA1, NULL);
 
 	/*
 	 * Corrupt the log section offset stored in the footer so that it points
@@ -278,7 +279,8 @@ void test_reftable_table__new_with_truncated_table(void)
 	struct reftable_table *table;
 	struct reftable_buf buf = REFTABLE_BUF_INIT;
 
-	cl_reftable_write_to_buf(&buf, refs, ARRAY_SIZE(refs), NULL, 0, NULL);
+	cl_reftable_write_to_buf(&buf, refs, ARRAY_SIZE(refs), NULL, 0,
+				 REFTABLE_HASH_SHA1, NULL);
 
 	/*
 	 * Truncate the table so that it is large enough to read the header, but
#28 lore raw

v2 cover receive-pack: use ODB transactions to stage object writes

In reply to Justin Tobler

Greetings,

This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.

Changes since V1:

- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling from odb_transaction_files_begin().
- odb_transaction_begin() now returns an error if there is already an inflight transaction pending instead of setting the out pointer to NULL.
- Updated odb_transaction_env() to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it from the_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.

Thanks,
-Justin

Justin Tobler (11):
object-file: rename files transaction prepare function
object-file: rename files transaction fsync function
object-file: embed transaction flush logic in commit function
object-file: drop check for inflight transactions
object-file: propagate files transaction errors odb/transaction: propagate begin errors odb/transaction: propagate commit errors odb/transaction: add transaction env interface odb/transaction: introduce ODB transaction flags builtin/receive-pack: drop redundant tmpdir env builtin/receive-pack: stage incoming objects via ODB transactions

 builtin/add.c            |   2 +-
 builtin/receive-pack.c   |  69 ++++++++---------
 builtin/unpack-objects.c |   2 +-
 builtin/update-index.c   |   2 +-
 cache-tree.c             |   7 +-
 object-file.c            | 159 +++++++++++++++++++++++++--------------
 object-file.h            |   8 +-
 odb/source-files.c       |   9 +--
 odb/source-inmemory.c    |   3 +-
 odb/source-loose.c       |   3 +-
 odb/source.h             |   9 ++-
 odb/transaction.c        |  34 +++++++--
 odb/transaction.h        |  60 ++++++++++++---
 read-cache.c             |   7 +-
 14 files changed, 244 insertions(+), 130 deletions(-)

Range-diff against v1:

 1:  9c14b219ad =  1:  9c14b219ad object-file: rename files transaction prepare function
 -:  ---------- >  2:  5703a9e93b object-file: rename files transaction fsync function
 -:  ---------- >  3:  4c37398ac8 object-file: embed transaction flush logic in commit function
 -:  ---------- >  4:  623c6b02ea object-file: drop check for inflight transactions
 2:  201f543692 !  5:  ca59176657 object-file: propagate files transaction errors

@@ Commit message
the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow
- callers to handle as needed.
+ callers to handle them as needed.

@@ object-file.c: static void odb_transaction_files_prepare(struct odb_transaction
+ return 0;
}

- static void fsync_loose_object_transaction(struct odb_transaction *base,
-@@ object-file.c: static void fsync_loose_object_transaction(struct odb_transaction *base,
- /*
- * Cleanup after batch-mode fsync_object_files.
- */
--static void flush_loose_object_transaction(struct odb_transaction_files *transaction) -+static int flush_loose_object_transaction(struct odb_transaction_files *transaction)
- {
- struct strbuf temp_path = STRBUF_INIT;
- struct tempfile *temp;
-
- if (!transaction->objdir)
-- return;
-+ return 0;
-
- /*
- * Issue a full hardware flush against a temporary file to ensure
-@@ object-file.c: static void flush_loose_object_transaction(struct odb_transaction_files *transac
- * Make the object files visible in the primary ODB after their data is
- * fully durable.
- */
-- tmp_objdir_migrate(transaction->objdir);
-+ if (tmp_objdir_migrate(transaction->objdir))
-+ return -1;
-+
- transaction->objdir = NULL;
-+
-+ return 0;
- }
-
- /* Finalize a file on disk, and close it. */
+ static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ object-file.c: int read_loose_object(struct repository *repo,
return ret;
}
@@ object-file.c: int read_loose_object(struct repository *repo,
{
struct odb_transaction_files *transaction =
container_of(base, struct odb_transaction_files, base);
+@@ object-file.c: static void odb_transaction_files_commit(struct odb_transaction *base)
+ * Make the object files visible in the primary ODB after their data is
+ * fully durable.
+ */
+- tmp_objdir_migrate(transaction->objdir);
++ if (tmp_objdir_migrate(transaction->objdir))
++ return -1;
++
+ transaction->objdir = NULL;
+ }

-- flush_loose_object_transaction(transaction);
-+ if (flush_loose_object_transaction(transaction))
-+ return -1;
flush_packfile_transaction(transaction);
+
+ return 0;
@@ object-file.c: int read_loose_object(struct repository *repo,
+ struct odb_transaction **out)
{
struct odb_transaction_files *transaction;
- struct object_database *odb = source->odb;
-
-- if (odb->transaction)
-- return NULL;
-+ if (odb->transaction) {
-+ *out = NULL;
-+ return 0;
-+ }

- transaction = xcalloc(1, sizeof(*transaction));
+@@ object-file.c: struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
transaction->base.source = source;
transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
@@ object-file.c: int read_loose_object(struct repository *repo,

## object-file.h ##
@@ object-file.h: struct odb_transaction;
- * Tell the object database to optimize for adding
* multiple objects. odb_transaction_files_commit must be called
- * to make new objects visible. If a transaction is already
-- * pending, NULL is returned.
-+ * pending, out is set to NULL.
+ * to make new objects visible.
*/
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source); +int odb_transaction_files_begin(struct odb_source *source,
@@ odb/source-files.c: static int odb_source_files_write_object_stream(struct odb_s

## odb/transaction.h ##
@@ odb/transaction.h: struct odb_transaction {
+ /* The ODB source the transaction is opened against. */
struct odb_source *source;

- /* The ODB source specific callback invoked to commit a transaction. */
+- /* The ODB source specific callback invoked to commit a transaction. */
- void (*commit)(struct odb_transaction *transaction);
++ /*
++ * The ODB source specific callback invoked to commit a transaction.
++ * Returns 0 on success, a negative error code otherwise.
++ */
+ int (*commit)(struct odb_transaction *transaction);

/*
3: 68cdd88ab5 ! 6: 717a1ce9a7 odb/transaction: propagate begin errors

    @@ Commit message
         convenience for callsites that do not need to handle errors
         explicitly.
     
    +    Note that `odb_transaction_begin()` now returns an error when the ODB
    +    already has an inflight transaction pending. ODB transaction call sites
    +    that may encounter an inflight transaction are updated to explicitly
    +    handle this case.
    +
         Signed-off-by: Justin Tobler <[email protected]>
     
      ## builtin/add.c ##
    @@ builtin/update-index.c: int cmd_update_index(int argc,
      			parseopt_state = parse_options_step(&ctx, options,
     
      ## cache-tree.c ##
    +@@ cache-tree.c: static int update_one(struct cache_tree *it,
    + 
    + int cache_tree_update(struct index_state *istate, int flags)
    + {
    ++	int inflight = !!the_repository->objects->transaction;
    + 	struct odb_transaction *transaction;
    + 	int skip, i;
    + 
     @@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)
      
      	trace_performance_enter();
      	trace2_region_enter("cache_tree", "update", istate->repo);
     -	transaction = odb_transaction_begin(the_repository->objects);
    -+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
    ++	if (!inflight)
    ++		odb_transaction_begin_or_die(the_repository->objects, &transaction);
      	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
      		       "", 0, &skip, flags);
    - 	odb_transaction_commit(transaction);
    +-	odb_transaction_commit(transaction);
    ++	if (!inflight)
    ++		odb_transaction_commit(transaction);
    + 	trace2_region_leave("cache_tree", "update", istate->repo);
    + 	trace_performance_leave("cache_tree_update");
    + 	if (i < 0)
     
      ## object-file.c ##
     @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
    @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
      		if (flags & INDEX_WRITE_OBJECT) {
      			struct object_database *odb = the_repository->objects;
     -			struct odb_transaction *transaction = odb_transaction_begin(odb);
    -+			struct odb_transaction *transaction;
    ++			struct odb_transaction *transaction = odb->transaction;
    ++			int inflight = !!transaction;
      
    -+			odb_transaction_begin_or_die(odb, &transaction);
    - 			ret = odb_transaction_write_object_stream(odb->transaction,
    +-			ret = odb_transaction_write_object_stream(odb->transaction,
    ++			if (!inflight)
    ++				odb_transaction_begin_or_die(odb, &transaction);
    ++			ret = odb_transaction_write_object_stream(transaction,
      								  &stream,
      								  xsize_t(st->st_size),
    + 								  oid);
    +-			odb_transaction_commit(transaction);
    ++			if (!inflight)
    ++				odb_transaction_commit(transaction);
    + 		} else {
    + 			ret = hash_blob_stream(&stream,
    + 					       the_repository->hash_algo, oid,
     
      ## odb/transaction.c ##
     @@
    @@ odb/transaction.c
     +int odb_transaction_begin(struct object_database *odb,
     +			  struct odb_transaction **out)
      {
    --	if (odb->transaction)
    --		return NULL;
     +	int ret;
    ++
    + 	if (odb->transaction)
    +-		return NULL;
    ++		return -1;
      
     -	odb_source_begin_transaction(odb->sources, &odb->transaction);
    -+	if (odb->transaction) {
    -+		*out = NULL;
    -+		return 0;
    -+	}
    - 
    --	return odb->transaction;
     +	ret = odb_source_begin_transaction(odb->sources, out);
     +	odb->transaction = *out;
    -+
    + 
    +-	return odb->transaction;
     +	return ret;
      }
      
    @@ odb/transaction.h: struct odb_transaction {
     + * Starts an ODB transaction and returns it via `out`. Subsequent objects are
     + * written to the transaction and not committed until odb_transaction_commit()
     + * is invoked on the transaction. Returns 0 on success and a negative value on
    -+ * error. If the ODB already has a pending transaction, `out` is set to NULL.
    ++ * error. Note that it is considered an error to start a new transaction if the
    ++ * ODB already has an inflight transaction pending.
       */
     -struct odb_transaction *odb_transaction_begin(struct object_database *odb);
     +int odb_transaction_begin(struct object_database *odb,
    @@ odb/transaction.h: struct odb_transaction {
       * Commits an ODB transaction making the written objects visible. If the
     
      ## read-cache.c ##
    +@@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,
    + 		       const struct pathspec *pathspec, char *ps_matched,
    + 		       int include_sparse, int flags, int ignored_too )
    + {
    ++	int inflight = !!repo->objects->transaction;
    + 	struct odb_transaction *transaction;
    + 	struct update_callback_data data;
    + 	struct rev_info rev;
     @@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,
      	 * This function is invoked from commands other than 'add', which
      	 * may not have their own transaction active.
      	 */
     -	transaction = odb_transaction_begin(repo->objects);
    -+	odb_transaction_begin_or_die(repo->objects, &transaction);
    ++	if (!inflight)
    ++		odb_transaction_begin_or_die(repo->objects, &transaction);
      	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
    - 	odb_transaction_commit(transaction);
    +-	odb_transaction_commit(transaction);
    ++	if (!inflight)
    ++		odb_transaction_commit(transaction);
      
    + 	release_revisions(&rev);
    + 	return !!data.add_errors;
 -:  ---------- >  7:  ff8e133965 odb/transaction: propagate commit errors
 5:  82302db9f0 !  8:  264ba94b83 odb/transaction: add transaction env interface
    @@ Commit message
     
         The ODB transaction backend is responsible for creating/managing its own
         staging area for writing objects. Other child processes spawned by Git
    -    may need to access to uncommitted objects or write new objects in the
    +    may need access to uncommitted objects or write new objects in the
         staging area though.
     
         Introduce `odb_transaction_env()` which is expected to provide the set
         of environment variables needed by a child process to access the
    -    transaction staging area.
    +    transaction's staging area.
     
         Signed-off-by: Justin Tobler <[email protected]>
     
      ## object-file.c ##
    +@@
    + #include "path.h"
    + #include "read-cache-ll.h"
    + #include "setup.h"
    ++#include "strvec.h"
    + #include "tempfile.h"
    + #include "tmp-objdir.h"
    + 
     @@ object-file.c: static int odb_transaction_files_commit(struct odb_transaction *base)
      	return 0;
      }
      
    -+static const char **odb_transaction_files_env(struct odb_transaction *base)
    ++static int odb_transaction_files_env(struct odb_transaction *base,
    ++				     struct strvec *env)
     +{
     +	struct odb_transaction_files *transaction =
     +		container_of(base, struct odb_transaction_files, base);
     +
     +	odb_transaction_files_prepare(&transaction->base);
    ++	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
     +
    -+	return tmp_objdir_env(transaction->objdir);
    ++	return 0;
     +}
     +
      int odb_transaction_files_begin(struct odb_source *source,
    @@ odb/transaction.c: int odb_transaction_write_object_stream(struct odb_transactio
      	return transaction->write_object_stream(transaction, stream, len, oid);
      }
     +
    -+const char **odb_transaction_env(struct odb_transaction *transaction)
    ++int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)
     +{
     +	if (!transaction)
    -+		return NULL;
    ++		return 0;
     +
    -+	return transaction->env(transaction);
    ++	return transaction->env(transaction, env);
     +}
     
      ## odb/transaction.h ##
    @@ odb/transaction.h: struct odb_transaction {
      				   struct object_id *oid);
     +
     +	/*
    -+	 * This callback is expected to return a NULL-terminated array of
    -+	 * environment variables that a child process should inherit so
    -+	 * that its object writes participate in the transaction. The
    -+	 * returned array is owned by the backend and remains valid until
    -+	 * the transaction ends. May return NULL when the backend does not
    -+	 * need to expose any state to child processes.
    ++	 * This callback is expected to populate the provided strvec with the
    ++	 * environment variables that a child process should inherit so that its
    ++	 * object writes participate in the transaction. Returns 0 on success, a
    ++	 * negative error code otherwise.
     +	 */
    -+	const char **(*env)(struct odb_transaction *transaction);
    ++	int (*env)(struct odb_transaction *transaction, struct strvec *env);
      };
      
      /*
    @@ odb/transaction.h: int odb_transaction_write_object_stream(struct odb_transactio
      					size_t len, struct object_id *oid);
      
     +/*
    -+ * Returns a NULL-terminated array of environment variables that a child
    ++ * Populates the provided strvec with the environment variables that a child
     + * process should inherit so that its object writes participate in the
    -+ * transaction, suitable for passing via child_process.env. Returns NULL if
    -+ * the transaction is NULL or the backend does not expose any state to child
    -+ * processes.
    ++ * transaction, suitable for using via child_process.env. Returns 0 on success,
    ++ * a negative error code otherwise. Note that, if the specified transaction is
    ++ * NULL, the function is a no-op and no error is returned.
     + */
    -+const char **odb_transaction_env(struct odb_transaction *transaction);
    ++int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env);
     +
      #endif

6: 0674e8a3d3 ! 9: 1e0a491ef2 builtin/receive-pack: stage incoming objects via ODB transactions
@@ Metadata
Author: Justin Tobler <[email protected]>

## Commit message ##
- builtin/receive-pack: stage incoming objects via ODB transactions
+ odb/transaction: introduce ODB transaction flags

- Objects received by git-receive-pack(1) are quarantined in a temporary
- "incoming" directory and migrated into the object database prior to the
- reference updates. The quarantine is currently managed through
- tmp_objdir directly. In a pluggable ODB future, how exactly an object
- gets written to a transaction may vary for a given ODB source. Refactor
- git-receive-pack(1) to use the ODB transaction interfaces to manage the
- object staging area in a more agnostic manner accordingly.
+ The temporary directory used by git-receive-pack(1) to write objects is
+ managed slightly differently than how it is done via ODB transactions:

- Note that the temporary directory created for git-receive-pack(1) is
- eagerly created and uses a different prefix name. This behavior is
- special cased in the "files" backend by having odb_transaction_begin()
- callers that require this behavior provide an ODB_TRANSACTION_RECEIVE
- flag.
+ - The temporary directory is eagerly created upfront, instead of
+ waiting for the first object write.
+
+ - The prefix name of the temporary directory is "incoming" instead of
+ "bulk-fsync".
+
+ In a subsequent commit, git-receive-pack(1) will use ODB transactions
+ instead of tmp_objdir directly. To provide a means to configure the
+ same transaction behavior, introduce enum odb_transaction_flags and
+ the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB
+ transactions using the "files" backend to be set up for
+ git-receive-pack(1). Transaction call sites are updated accordingly to
+ provide the required flag parameter.

@@ builtin/add.c: int cmd_add(int argc,
ps_matched = xcalloc(pathspec.nr, 1);
if (add_renormalize)

- ## builtin/receive-pack.c ##
-@@ builtin/receive-pack.c: static enum {
- } use_keepalive;
- static int keepalive_in_sec = 5;
-
--static struct tmp_objdir *tmp_objdir;
--
- static struct proc_receive_ref {
- unsigned int want_add:1,
- want_delete:1,
-@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
- strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
- }
-
-- if (tmp_objdir)
-- strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
-+ if (the_repository->objects->transaction)
-+ strvec_pushv(&opt.env, odb_transaction_env(the_repository->objects->transaction));
-
- prepare_push_cert_sha1(&opt);
-
-@@ builtin/receive-pack.c: static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
- !delayed_reachability_test(si, i))
- oid_array_append(&extra, &si->shallow->oid[i]);
-
-- opt.env = tmp_objdir_env(tmp_objdir);
-+ opt.env = odb_transaction_env(the_repository->objects->transaction);
- setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
- if (check_connected(command_singleton_iterator, cmd, &opt)) {
- rollback_shallow_file(the_repository, &shallow_lock);
-@@ builtin/receive-pack.c: static void set_connectivity_errors(struct command *commands,
- /* to be checked in update_shallow_ref() */
- continue;
-
-- opt.env = tmp_objdir_env(tmp_objdir);
-+ opt.env = odb_transaction_env(the_repository->objects->transaction);
- if (!check_connected(command_singleton_iterator, &singleton,
- &opt))
- continue;
-@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
- data.si = si;
- opt.err_fd = err_fd;
- opt.progress = err_fd && !quiet;
-- opt.env = tmp_objdir_env(tmp_objdir);
-+ opt.env = odb_transaction_env(the_repository->objects->transaction);
- opt.exclude_hidden_refs_section = "receive";
-
- if (check_connected(iterate_receive_command_list, &data, &opt))
-@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
- * Now we'll start writing out refs, which means the objects need
- * to be in their final positions so that other processes can see them.
- */
-- if (tmp_objdir_migrate(tmp_objdir) < 0) {
-+ if (odb_transaction_commit(the_repository->objects->transaction)) {
- for (cmd = commands; cmd; cmd = cmd->next) {
- if (!cmd->error_string)
- cmd->error_string = "unable to migrate objects to permanent storage";
- }
- return;
- }
-- tmp_objdir = NULL;
-
- check_aliased_updates(commands);
-
-@@ builtin/receive-pack.c: static void push_header_arg(struct strvec *args, struct pack_header *hdr)
- ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
- }
-
--static const char *unpack(int err_fd, struct shallow_info *si)
-+static const char *unpack(int err_fd, struct shallow_info *si,
-+ struct odb_transaction *transaction)
- {
- struct pack_header hdr;
- const char *hdr_err;
-@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
- strvec_push(&child.args, alt_shallow_file);
- }
-
-- tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-- if (!tmp_objdir) {
-- if (err_fd > 0)
-- close(err_fd);
-- return "unable to create temporary object directory";
-- }
-- strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
--
-- /*
-- * Normally we just pass the tmp_objdir environment to the child
-- * processes that do the heavy lifting, but we may need to see these
-- * objects ourselves to set up shallow information.
-- */
-- tmp_objdir_add_as_alternate(tmp_objdir);
-+ strvec_pushv(&child.env, odb_transaction_env(transaction));
-
- if (ntohl(hdr.hdr_entries) < unpack_limit) {
- strvec_push(&child.args, "unpack-objects");
-@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
- return NULL;
- }
-
--static const char *unpack_with_sideband(struct shallow_info *si)
-+static const char *unpack_with_sideband(struct shallow_info *si,
-+ struct odb_transaction *transaction)
- {
- struct async muxer;
- const char *ret;
-
- if (!use_sideband)
-- return unpack(0, si);
-+ return unpack(0, si, transaction);
-
- use_keepalive = KEEPALIVE_AFTER_NUL;
- memset(&muxer, 0, sizeof(muxer));
-@@ builtin/receive-pack.c: static const char *unpack_with_sideband(struct shallow_info *si)
- if (start_async(&muxer))
- return NULL;
-
-- ret = unpack(muxer.in, si);
-+ ret = unpack(muxer.in, si, transaction);
-
- finish_async(&muxer);
- return ret;
-@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
- struct oid_array ref = OID_ARRAY_INIT;
- struct shallow_info si;
- struct packet_reader reader;
-+ struct odb_transaction *transaction = NULL;
-
- struct option options[] = {
- OPT__QUIET(&quiet, N_("quiet")),
-@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
- if (!si.nr_ours && !si.nr_theirs)
- shallow_update = 0;
- if (!delete_only(commands)) {
-- unpack_status = unpack_with_sideband(&si);
-+ if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
-+ unpack_status = "unable to start ODB transaction";
-+ else
-+ unpack_status = unpack_with_sideband(&si, transaction);
- update_shallow_info(commands, &si, &ref);
- }
- use_keepalive = KEEPALIVE_ALWAYS;
-
## builtin/unpack-objects.c ##
@@ builtin/unpack-objects.c: static void unpack_all(void)
progress = start_progress(the_repository,
@@ builtin/update-index.c: int cmd_update_index(int argc,

## cache-tree.c ##
@@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)
-
trace_performance_enter();
trace2_region_enter("cache_tree", "update", istate->repo);
-- odb_transaction_begin_or_die(the_repository->objects, &transaction);
-+ odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
+ if (!inflight)
+- odb_transaction_begin_or_die(the_repository->objects, &transaction);
++ odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
"", 0, &skip, flags);
- odb_transaction_commit(transaction);
+ if (!inflight)

## object-file.c ##
@@ object-file.c: struct odb_transaction_files {
@@ object-file.c: static int odb_transaction_files_prepare(struct odb_transaction *
return -1;

@@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
- struct object_database *odb = the_repository->objects;
- struct odb_transaction *transaction;
+ int inflight = !!transaction;

-- odb_transaction_begin_or_die(odb, &transaction);
-+ odb_transaction_begin_or_die(odb, &transaction, 0);
- ret = odb_transaction_write_object_stream(odb->transaction,
+ if (!inflight)
+- odb_transaction_begin_or_die(odb, &transaction);
++ odb_transaction_begin_or_die(odb, &transaction, 0);
+ ret = odb_transaction_write_object_stream(transaction,
&stream, xsize_t(st->st_size),
-@@ object-file.c: static const char **odb_transaction_files_env(struct odb_transaction *base) +@@ object-file.c: static int odb_transaction_files_env(struct odb_transaction *base,
}

int odb_transaction_files_begin(struct odb_source *source,
@@ object-file.c: static const char **odb_transaction_files_env(struct odb_transact
+ enum odb_transaction_flags flags)
{
struct odb_transaction_files *transaction;
- struct object_database *odb = source->odb;
+
@@ object-file.c: int odb_transaction_files_begin(struct odb_source *source,
transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
@@ object-file.c: int odb_transaction_files_begin(struct odb_source *source,
+ if (flags & ODB_TRANSACTION_RECEIVE) {
+ /*
+ * ODB transactions for git-receive-pack(1) eagerly create a
-+ * temporary directory and use a different prefix.
++ * temporary directory and use a different temporary directory
++ * prefix.
++ *
++ * NEEDSWORK: This transaction flag is only used by the "files"
++ * backend to special case temporary directory set up and
++ * handling. Ideally transaction users should not have to care
++ * though. To avoid this, we could eagerly create the temporary
++ * directory and use the same prefix name for all transactions.
+ */
+ transaction->prefix = "incoming";
+ if (odb_transaction_files_prepare(&transaction->base)) {
@@ object-file.h
/* The maximum size for an object header. */
#define MAX_HEADER_LEN 32
@@ object-file.h: struct odb_transaction;
- * pending, out is set to NULL.
+ * to make new objects visible.
*/
int odb_transaction_files_begin(struct odb_source *source,
- struct odb_transaction **out);
@@ odb/transaction.c
{
int ret;

-@@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
- return 0;
- }
+ if (odb->transaction)
+ return -1;

- ret = odb_source_begin_transaction(odb->sources, out);
+ ret = odb_source_begin_transaction(odb->sources, out, flags);
@@ odb/transaction.h
/*
* A transaction may be started for an object database prior to writing new
@@ odb/transaction.h: struct odb_transaction {
- const char **(*env)(struct odb_transaction *transaction);
+ int (*env)(struct odb_transaction *transaction, struct strvec *env);
};

++/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
++ /* Configures the transaction for use with git-receive-pack(1). */
+ ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
@@ odb/transaction.h: struct odb_transaction {
* Starts an ODB transaction and returns it via out. Subsequent objects are
* written to the transaction and not committed until odb_transaction_commit()
@@ odb/transaction.h: struct odb_transaction {
- * error. If the ODB already has a pending transaction, out is set to NULL.
+ * ODB already has an inflight transaction pending.
*/
int odb_transaction_begin(struct object_database *odb,
- struct odb_transaction **out);
@@ odb/transaction.h: struct odb_transaction {

## read-cache.c ##
@@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,
- * This function is invoked from commands other than 'add', which
* may not have their own transaction active.
*/
-- odb_transaction_begin_or_die(repo->objects, &transaction);
-+ odb_transaction_begin_or_die(repo->objects, &transaction, 0);
+ if (!inflight)
+- odb_transaction_begin_or_die(repo->objects, &transaction);
++ odb_transaction_begin_or_die(repo->objects, &transaction, 0);
run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
- odb_transaction_commit(transaction);
-
+ if (!inflight)
+ odb_transaction_commit(transaction);

 -:  ---------- > 10:  6c8d878349 builtin/receive-pack: drop redundant tmpdir env
 4:  e9303f9b08 ! 11:  8db95fef56 odb/transaction: propagate commit errors

@@ Metadata
Author: Justin Tobler <[email protected]>

## Commit message ##
- odb/transaction: propagate commit errors
+ builtin/receive-pack: stage incoming objects via ODB transactions

- When odb_transaction_commit() is invoked, the return value of the
- backend commit callback is silently discarded. A backend has no way
- to signal that committing failed, such as when the "files" backend
- cannot migrate its temporary object directory into the permanent
- ODB.
+ Objects received by git-receive-pack(1) are quarantined in a temporary
+ "incoming" directory and migrated into the object database prior to the
+ reference updates. The quarantine is currently managed through
+ tmp_objdir directly. In a pluggable ODB future, how exactly an object
+ gets written to a transaction may vary for a given ODB source. Refactor
+ git-receive-pack(1) to use the ODB transaction interfaces to manage the
+ object staging area in a more agnostic manner accordingly.

- In a subsequent commit, git-receive-pack(1) starts using ODB transaction
- to stage objects and consequently cares about such failures so it can
- handle the error appropriately. Change the commit callback signature to
- return an int error code and have odb_transaction_commit() forward it
- accordingly.
+ Note that the ODB transaction is now responsible for managing the
+ primary and alternate ODBs for the repository. One small change as a
+ result is that the temporary directory is now applied as the primary ODB
+ in the main process instead of an alternate. This does not change
+ anything for git-receive-pack(1) though because it only needs access to
+ the newly written objects and doesn't care how exactly it is set up.

- ## odb/transaction.c ##
-@@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
- return ret;
+ ## builtin/receive-pack.c ##
+@@
+ #include "sigchain.h"
+ #include "string-list.h"
+ #include "strvec.h"
+-#include "tmp-objdir.h"
+ #include "trace.h"
+ #include "trace2.h"
+ #include "version.h"
+@@ builtin/receive-pack.c: static enum {
+ } use_keepalive;
+ static int keepalive_in_sec = 5;
+
+-static struct tmp_objdir *tmp_objdir;
+-
+ static struct proc_receive_ref {
+ unsigned int want_add:1,
+ want_delete:1,
+@@ builtin/receive-pack.c: static void receive_hook_feed_state_free(void *data)
+ static int run_receive_hook(struct command *commands,
+ const char *hook_name,
+ int skip_broken,
++ struct odb_transaction *transaction,
+ const struct string_list *push_options)
+ {
+ struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
+ strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
+ }
+
+- if (tmp_objdir)
+- strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
++ if (transaction)
++ odb_transaction_env(transaction, &opt.env);
+
+ prepare_push_cert_sha1(&opt);
+
+@@ builtin/receive-pack.c: static const struct object_id *command_singleton_iterator(void *cb_data)
}

--void odb_transaction_commit(struct odb_transaction *transaction) -+int odb_transaction_commit(struct odb_transaction *transaction)
+ static void set_connectivity_errors(struct command *commands,
+- struct shallow_info *si)
++ struct shallow_info *si,
++ struct odb_transaction *transaction)
{
-+ int ret;
+ struct command *cmd;
+
+ for (cmd = commands; cmd; cmd = cmd->next) {
+ struct command *singleton = cmd;
+ struct check_connected_options opt = CHECK_CONNECTED_INIT;
++ struct strvec env = STRVEC_INIT;
+
+ if (shallow_update && si->shallow_ref[cmd->index])
+ /* to be checked in update_shallow_ref() */
+ continue;
+
+- opt.env = tmp_objdir_env(tmp_objdir);
++ odb_transaction_env(transaction, &env);
++ opt.env = env.v;
+
- if (!transaction)
-- return;
-+ return 0;
+ if (!check_connected(command_singleton_iterator, &singleton,
+ &opt))
+ continue;

- /*
- * Ensure the transaction ending matches the pending transaction.
- */
- ASSERT(transaction == transaction->source->odb->transaction);
+ cmd->error_string = "missing necessary objects";
++
++ strvec_clear(&env);
+ }
+ }
+
+@@ builtin/receive-pack.c: static void execute_commands_atomic(struct command *commands,
+ static void execute_commands(struct command *commands,
+ const char *unpacker_error,
+ struct shallow_info *si,
++ struct odb_transaction *transaction,
+ const struct string_list *push_options)
+ {
+ struct check_connected_options opt = CHECK_CONNECTED_INIT;
+@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
+ }

-- transaction->commit(transaction);
-+ ret = transaction->commit(transaction);
-+ if (ret)
-+ return ret;
+ if (!skip_connectivity_check) {
++ struct strvec env = STRVEC_INIT;
+
- transaction->source->odb->transaction = NULL;
- free(transaction);
+ if (use_sideband) {
+ memset(&muxer, 0, sizeof(muxer));
+ muxer.proc = copy_to_sideband;
+@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
+ data.si = si;
+ opt.err_fd = err_fd;
+ opt.progress = err_fd && !quiet;
+- opt.env = tmp_objdir_env(tmp_objdir);
++ odb_transaction_env(transaction, &env);
++ opt.env = env.v;
+ opt.exclude_hidden_refs_section = "receive";
+
+ if (check_connected(iterate_receive_command_list, &data, &opt))
+- set_connectivity_errors(commands, si);
++ set_connectivity_errors(commands, si, transaction);
+
+ if (use_sideband)
+ finish_async(&muxer);
+
-+ return 0;
++ strvec_clear(&env);
+ }
+
+ reject_updates_to_hidden(commands);
+@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
+ }
+ }
+
+- if (run_receive_hook(commands, "pre-receive", 0, push_options)) {
++ if (run_receive_hook(commands, "pre-receive", 0, transaction, push_options)) {
+ for (cmd = commands; cmd; cmd = cmd->next) {
+ if (!cmd->error_string)
+ cmd->error_string = "pre-receive hook declined";
+@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
+ * Now we'll start writing out refs, which means the objects need
+ * to be in their final positions so that other processes can see them.
+ */
+- if (tmp_objdir_migrate(tmp_objdir) < 0) {
++ if (odb_transaction_commit(transaction)) {
+ for (cmd = commands; cmd; cmd = cmd->next) {
+ if (!cmd->error_string)
+ cmd->error_string = "unable to migrate objects to permanent storage";
+ }
+ return;
+ }
+- tmp_objdir = NULL;
+
+ check_aliased_updates(commands);
+
+@@ builtin/receive-pack.c: static void push_header_arg(struct strvec *args, struct pack_header *hdr)
+ ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
}

- int odb_transaction_write_object_stream(struct odb_transaction *transaction,
-
- ## odb/transaction.h ##
-@@ odb/transaction.h: static inline void odb_transaction_begin_or_die(struct object_database *odb,
- * Commits an ODB transaction making the written objects visible. If the
- * specified transaction is NULL, the function is a no-op.
- */
--void odb_transaction_commit(struct odb_transaction *transaction);
-+int odb_transaction_commit(struct odb_transaction *transaction);
-
- /*
- * Writes the object in the provided stream into the transaction. The resulting +-static const char *unpack(int err_fd, struct shallow_info *si)
++static const char *unpack(int err_fd, struct shallow_info *si,
++ struct odb_transaction *transaction)
+ {
+ struct pack_header hdr;
+ const char *hdr_err;
+@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
+ strvec_push(&child.args, alt_shallow_file);
+ }
+
+- tmp_objdir = tmp_objdir_create(the_repository, "incoming");
+- if (!tmp_objdir) {
+- if (err_fd > 0)
+- close(err_fd);
+- return "unable to create temporary object directory";
+- }
+- strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
+-
+- /*
+- * Normally we just pass the tmp_objdir environment to the child
+- * processes that do the heavy lifting, but we may need to see these
+- * objects ourselves to set up shallow information.
+- */
+- tmp_objdir_add_as_alternate(tmp_objdir);
++ odb_transaction_env(transaction, &child.env);
+
+ if (ntohl(hdr.hdr_entries) < unpack_limit) {
+ strvec_push(&child.args, "unpack-objects");
+@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
+ return NULL;
+ }
+
+-static const char *unpack_with_sideband(struct shallow_info *si)
++static const char *unpack_with_sideband(struct shallow_info *si,
++ struct odb_transaction *transaction)
+ {
+ struct async muxer;
+ const char *ret;
+
+ if (!use_sideband)
+- return unpack(0, si);
++ return unpack(0, si, transaction);
+
+ use_keepalive = KEEPALIVE_AFTER_NUL;
+ memset(&muxer, 0, sizeof(muxer));
+@@ builtin/receive-pack.c: static const char *unpack_with_sideband(struct shallow_info *si)
+ if (start_async(&muxer))
+ return NULL;
+
+- ret = unpack(muxer.in, si);
++ ret = unpack(muxer.in, si, transaction);
+
+ finish_async(&muxer);
+ return ret;
+@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
+ struct oid_array ref = OID_ARRAY_INIT;
+ struct shallow_info si;
+ struct packet_reader reader;
++ struct odb_transaction *transaction = NULL;
+
+ struct option options[] = {
+ OPT__QUIET(&quiet, N_("quiet")),
+@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
+ if (!si.nr_ours && !si.nr_theirs)
+ shallow_update = 0;
+ if (!delete_only(commands)) {
+- unpack_status = unpack_with_sideband(&si);
++ if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
++ unpack_status = "unable to start object transaction";
++ else
++ unpack_status = unpack_with_sideband(&si, transaction);
+ update_shallow_info(commands, &si, &ref);
+ }
+ use_keepalive = KEEPALIVE_ALWAYS;
+- execute_commands(commands, unpack_status, &si,
++ execute_commands(commands, unpack_status, &si, transaction,
+ &push_options);
+ delete_tempfile(&pack_lockfile);
+ sigchain_push(SIGPIPE, SIG_IGN);
+@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
+ else if (report_status)
+ report(commands, unpack_status);
+ sigchain_pop(SIGPIPE);
+- run_receive_hook(commands, "post-receive", 1,
++ run_receive_hook(commands, "post-receive", 1, NULL,
+ &push_options);
+ run_update_post_hook(commands);
+ free_commands(commands);

base-commit: ab776a62a78576513ee121424adb19597fbb7613

Signature · 1 line
2.55.0.122.gf85a7e6620
#29 lore raw

PATCH v2 1/11 object-file: rename files transaction prepare function

The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.

Rename the function to odb_transaction_files_prepare() accordingly.

---
 object-file.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
object-file.c +3 -3
index e3d92bbda2..a3eb8d71dd 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void prepare_loose_object_transaction(struct odb_transaction *base)
+static void odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf filename = STRBUF_INIT;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	odb_loose_path(loose, &filename, oid);
 
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,
 	int hdrlen;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
Signature · 1 line
2.55.0.122.gf85a7e6620
#30 lore raw

PATCH v2 2/11 object-file: rename files transaction fsync function

In reply to Justin Tobler

When writing an object to a "files" ODB transaction, a full hardware flush is not initially performed during the fsync in fsync_loose_object_transaction() and instead delayed until the transaction is later committed.

To be more consistent with other "files" ODB transaction helpers, rename the function to odb_transaction_files_fsync() accordingly. The conditional in the helper is also slightly restructured to improve clarity to readers.

---
 object-file.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
object-file.c +10 -6
index a3eb8d71dd..d68824bb44 100644
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
 }
 
-static void fsync_loose_object_transaction(struct odb_transaction *base,
-					   int fd, const char *filename)
+static void odb_transaction_files_fsync(struct odb_transaction *base,
+					int fd, const char *filename)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
 
+	if (!transaction || !transaction->objdir) {
+		fsync_or_die(fd, filename);
+		return;
+	}
+
 	/*
 	 * If we have an active ODB transaction, we issue a call that
 	 * cleans the filesystem page cache but avoids a hardware flush
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 	 * before renaming the objects to their final names as part of
 	 * flush_batch_fsync.
 	 */
-	if (!transaction || !transaction->objdir ||
-	    git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
+	if (git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
 		if (errno == ENOSYS)
 			warning(_("core.fsyncMethod = batch is unsupported on this platform"));
 		fsync_or_die(fd, filename);
@@ -553,7 +557,7 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
 	 * that all objects are durable before any renames occur. The code in
-	 * fsync_loose_object_transaction has already issued a writeout
+	 * odb_transaction_files_fsync has already issued a writeout
 	 * request, but it has not flushed any writeback cache in the storage
 	 * hardware or any filesystem logs. This fsync call acts as a barrier
 	 * to ensure that the data in each new object file is durable before
@@ -582,7 +586,7 @@ static void close_loose_object(struct odb_source_loose *loose,
 		goto out;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		fsync_loose_object_transaction(loose->base.odb->transaction, fd, filename);
+		odb_transaction_files_fsync(loose->base.odb->transaction, fd, filename);
 	else if (fsync_object_files > 0)
 		fsync_or_die(fd, filename);
 	else
Signature · 1 line
2.55.0.122.gf85a7e6620
#31 lore raw

Re: [PATCH v2 02/11] object-file: rename files transaction fsync function

Quoted 10 lines · Justin Tobler
object-file.c +0 -2
index a3eb8d71dd..d68824bb44 100644
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
 }
 
-static void fsync_loose_object_transaction(struct odb_transaction *base,
-					   int fd, const char *filename)

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 530

Quoted 10 lines
object-file.c +6 -0
+static void odb_transaction_files_fsync(struct odb_transaction *base,
+					int fd, const char *filename)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
 
+	if (!transaction || !transaction->objdir) {
+		fsync_or_die(fd, filename);
+		return;
+	}

The change results in a tiny bit of duplication, but I agree that it's easier to reason about.

object-file.c static void fsync_loose_object_transaction(struct odb_transaction *base,

Quoted 1 line
object-file.c +0 -0
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,

Somewhat funny that the diff renderer decided to put the hunk header here instead of showing the single line that's now missing from the diff.

object-file.c static void fsync_loose_object_transaction(struct odb_transaction *base, · line 542

Quoted 9 lines
object-file.c +1 -2
 	 * before renaming the objects to their final names as part of
 	 * flush_batch_fsync.
 	 */
-	if (!transaction || !transaction->objdir ||
-	    git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
+	if (git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
 		if (errno == ENOSYS)
 			warning(_("core.fsyncMethod = batch is unsupported on this platform"));
 		fsync_or_die(fd, filename);

Thanks!

Patrick

#32 lore raw

PATCH v2 3/11 object-file: embed transaction flush logic in commit function

In reply to Justin Tobler

When a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary. In a subsequent commit, the temporary directory is also used to write packfiles.

Instead of maintaining a separate helper function, embed the logic to flush and migrate the temporary directory directly into odb_transaction_files_commit().

---
 object-file.c | 64 ++++++++++++++++++++++-----------------------------
 1 file changed, 28 insertions(+), 36 deletions(-)
object-file.c +28 -36
index d68824bb44..33bd6c6810 100644
@@ -543,41 +543,6 @@ static void odb_transaction_files_fsync(struct odb_transaction *base,
 	}
 }
 
-/*
- * Cleanup after batch-mode fsync_object_files.
- */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
-{
-	struct strbuf temp_path = STRBUF_INIT;
-	struct tempfile *temp;
-
-	if (!transaction->objdir)
-		return;
-
-	/*
-	 * Issue a full hardware flush against a temporary file to ensure
-	 * that all objects are durable before any renames occur. The code in
-	 * odb_transaction_files_fsync has already issued a writeout
-	 * request, but it has not flushed any writeback cache in the storage
-	 * hardware or any filesystem logs. This fsync call acts as a barrier
-	 * to ensure that the data in each new object file is durable before
-	 * the final name is visible.
-	 */
-	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
-		    repo_get_object_directory(transaction->base.source->odb->repo));
-	temp = xmks_tempfile(temp_path.buf);
-	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
-	delete_tempfile(&temp);
-	strbuf_release(&temp_path);
-
-	/*
-	 * Make the object files visible in the primary ODB after their data is
-	 * fully durable.
-	 */
-	tmp_objdir_migrate(transaction->objdir);
-	transaction->objdir = NULL;
-}
-
 /* Finalize a file on disk, and close it. */
 static void close_loose_object(struct odb_source_loose *loose,
 			       int fd, const char *filename)
@@ -1679,7 +1644,34 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (transaction->objdir) {
+		struct strbuf temp_path = STRBUF_INIT;
+		struct tempfile *temp;
+
+		/*
+		 * Issue a full hardware flush against a temporary file to ensure
+		 * that all objects are durable before any renames occur. The code in
+		 * odb_transaction_files_fsync has already issued a writeout
+		 * request, but it has not flushed any writeback cache in the storage
+		 * hardware or any filesystem logs. This fsync call acts as a barrier
+		 * to ensure that the data in each new object file is durable before
+		 * the final name is visible.
+		 */
+		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
+			    repo_get_object_directory(transaction->base.source->odb->repo));
+		temp = xmks_tempfile(temp_path.buf);
+		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
+		delete_tempfile(&temp);
+		strbuf_release(&temp_path);
+
+		/*
+		 * Make the object files visible in the primary ODB after their data is
+		 * fully durable.
+		 */
+		tmp_objdir_migrate(transaction->objdir);
+		transaction->objdir = NULL;
+	}
+
 	flush_packfile_transaction(transaction);
 }
 
Signature · 1 line
2.55.0.122.gf85a7e6620
#33 lore raw

Re: [PATCH v2 03/11] object-file: embed transaction flush logic in commit function

Quoted 9 lines · Justin Tobler

When a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary. In a subsequent commit, the temporary directory is also used to write packfiles.

Instead of maintaining a separate helper function, embed the logic to flush and migrate the temporary directory directly into odb_transaction_files_commit().

The change itself looks simple enough, but this makes me wonder why we want to change this. Like, what subsequent step does this enable that would otherwise be harder to do?

Maybe this will be answered by a subsequent commit.

Patrick

#34 lore raw

Re: [PATCH v2 03/11] object-file: embed transaction flush logic in commit function

Quoted 7 lines · Patrick Steinhardt
Quoted 6 lines · Justin Tobler

When a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary. In a subsequent commit, the temporary directory is also used to write packfiles.

Quoted 7 lines
Quoted 3 lines

Instead of maintaining a separate helper function, embed the logic to flush and migrate the temporary directory directly into odb_transaction_files_commit().

The change itself looks simple enough, but this makes me wonder why we want to change this. Like, what subsequent step does this enable that would otherwise be harder to do?

Originally, I was planning to rename both {fsync,flush}_loose_object_transaction() to odb_transaction_files_{fsync,flush}() respectively. For the flush helper though, it's doing more than just "flushing" the temporary directory files; it's also migrating the objects to the primary ODB and reconfiguring the repository ODB sources accordingly.

This is really what you think of happening during the commit phase. Consequently, it made more sense to me organizationally to just embed the logic odb_transaction_files_commit().

Quoted 1 line

Maybe this will be answered by a subsequent commit.

Will update the commit message to properly explain this.

-Justin

#35 lore raw

PATCH v2 4/11 object-file: drop check for inflight transactions

In reply to Justin Tobler

ODB transactions are started via odb_transaction_begin() and contain validation to avoid starting multiple transactions at the same time. The "files" backend also has the same logic, but is redundant due to the generic layer already handling it. Drop this validation from the "files" backend accordingly.

---
 object-file.c | 4 ----
 object-file.h | 3 +--
 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -4
index 33bd6c6810..e51389833a 100644
@@ -1678,10 +1678,6 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 {
 	struct odb_transaction_files *transaction;
-	struct object_database *odb = source->odb;
-
-	if (odb->transaction)
-		return NULL;
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
object-file.h +1 -2
index 528c4e6e69..ea43d818f0 100644
@@ -194,8 +194,7 @@ struct odb_transaction;
 /*
  * Tell the object database to optimize for adding
  * multiple objects. odb_transaction_files_commit must be called
- * to make new objects visible. If a transaction is already
- * pending, NULL is returned.
+ * to make new objects visible.
  */
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
 
Signature · 1 line
2.55.0.122.gf85a7e6620
#36 lore raw

Re: [PATCH v2 04/11] object-file: drop check for inflight transactions

Quoted 5 lines · Justin Tobler

ODB transactions are started via odb_transaction_begin() and contain validation to avoid starting multiple transactions at the same time. The "files" backend also has the same logic, but is redundant due to the generic layer already handling it. Drop this validation from the "files" backend accordingly.

Makes sense, and it fixes a layering violation: in the best case, a source only has to care about itself and not about the owning object database. Managing object-database-level state should be done by the object database itself.

Patrick

#37 lore raw

PATCH v2 5/11 object-file: propagate files transaction errors

In reply to Justin Tobler

The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle them as needed.

---
 object-file.c      | 26 ++++++++++++++++++--------
 object-file.h      |  3 ++-
 odb/source-files.c |  6 +-----
 odb/transaction.h  |  7 +++++--
 4 files changed, 26 insertions(+), 16 deletions(-)
object-file.c +18 -8
index e51389833a..64cb874fe7 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;
+
+	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+
+	return 0;
 }
 
 static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ -1639,7 +1643,7 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 		 * Make the object files visible in the primary ODB after their data is
 		 * fully durable.
 		 */
-		tmp_objdir_migrate(transaction->objdir);
+		if (tmp_objdir_migrate(transaction->objdir))
+			return -1;
+
 		transaction->objdir = NULL;
 	}
 
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1683,6 +1692,7 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
-	return &transaction->base;
+	return 0;
 }
object-file.h +2 -1
index ea43d818f0..1a023226ac 100644
@@ -196,6 +196,7 @@ struct odb_transaction;
  * multiple objects. odb_transaction_files_commit must be called
  * to make new objects visible.
  */
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out);
 
 #endif /* OBJECT_FILE_H */
odb/source-files.c +1 -5
index 5bdd042922..2545bd81d4 100644
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 static int odb_source_files_begin_transaction(struct odb_source *source,
 					      struct odb_transaction **out)
 {
-	struct odb_transaction *tx = odb_transaction_files_begin(source);
-	if (!tx)
-		return -1;
-	*out = tx;
-	return 0;
+	return odb_transaction_files_begin(source, out);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
odb/transaction.h +5 -2
index 854fda06f5..d52f0533ce 100644
@@ -16,8 +16,11 @@ struct odb_transaction {
 	/* The ODB source the transaction is opened against. */
 	struct odb_source *source;
 
-	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	/*
+	 * The ODB source specific callback invoked to commit a transaction.
+	 * Returns 0 on success, a negative error code otherwise.
+	 */
+	int (*commit)(struct odb_transaction *transaction);
 
 	/*
 	 * This callback is expected to write the given object stream into
Signature · 1 line
2.55.0.122.gf85a7e6620
#38 lore raw

Re: [PATCH v2 05/11] object-file: propagate files transaction errors

Quoted 6 lines · Justin Tobler
object-file.c +0 -0
index e51389833a..64cb874fe7 100644
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518

Quoted 10 lines
object-file.c +3 -3
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;

As far as I can see we don't report any errors as part of tmp_objdir_create(), so we should probably print an error here.

object-file.c static void odb_transaction_files_commit(struct odb_transaction *base) · line 1677

Quoted 8 lines
object-file.c +3 -1
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 		 * Make the object files visible in the primary ODB after their data is
 		 * fully durable.
 		 */
-		tmp_objdir_migrate(transaction->objdir);
+		if (tmp_objdir_migrate(transaction->objdir))
+			return -1;
+

Likewise.

Patrick

#39 lore raw

Re: [PATCH v2 05/11] object-file: propagate files transaction errors

Quoted 21 lines · Patrick Steinhardt
Quoted 6 lines · Justin Tobler
object-file.c +0 -0
index e51389833a..64cb874fe7 100644
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.

object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518

Quoted 10 lines
object-file.c +3 -3
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return -1;

As far as I can see we don't report any errors as part of tmp_objdir_create(), so we should probably print an error here.

Quoted 10 lines

object-file.c static void odb_transaction_files_commit(struct odb_transaction *base) · line 1677

Quoted 8 lines
object-file.c +3 -1
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 		 * Make the object files visible in the primary ODB after their data is
 		 * fully durable.
 		 */
-		tmp_objdir_migrate(transaction->objdir);
+		if (tmp_objdir_migrate(transaction->objdir))
+			return -1;
+

Likewise.

Ya, printing some error messages seems like a good idea here. Will do so in the next version.

-Justin

#40 lore raw

PATCH v2 6/11 odb/transaction: propagate begin errors

In reply to Justin Tobler

When odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.

Note that odb_transaction_begin() now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.

---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  7 +++++--
 object-file.c            | 10 +++++++---
 odb/transaction.c        | 12 ++++++++----
 odb/transaction.h        | 20 ++++++++++++++++----
 read-cache.c             |  7 +++++--
 8 files changed, 44 insertions(+), 18 deletions(-)
builtin/add.c +1 -1
index c859f66519..3d5d9cfdb9 100644
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
builtin/unpack-objects.c +1 -1
index f3849bb654..d0136cdd99 100644
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
builtin/update-index.c +1 -1
index 3d6646c318..17f3ea284c 100644
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +5 -2
index 184f7e2635..8eec1d4d52 100644
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,
 
 int cache_tree_update(struct index_state *istate, int flags)
 {
+	int inflight = !!the_repository->objects->transaction;
 	struct odb_transaction *transaction;
 	int skip, i;
 
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 	trace2_region_leave("cache_tree", "update", istate->repo);
 	trace_performance_leave("cache_tree_update");
 	if (i < 0)
object-file.c +7 -3
index 64cb874fe7..cd1aa36462 100644
@@ -1354,13 +1354,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 
 		if (flags & INDEX_WRITE_OBJECT) {
 			struct object_database *odb = the_repository->objects;
-			struct odb_transaction *transaction = odb_transaction_begin(odb);
+			struct odb_transaction *transaction = odb->transaction;
+			int inflight = !!transaction;
 
-			ret = odb_transaction_write_object_stream(odb->transaction,
+			if (!inflight)
+				odb_transaction_begin_or_die(odb, &transaction);
+			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
 								  oid);
-			odb_transaction_commit(transaction);
+			if (!inflight)
+				odb_transaction_commit(transaction);
 		} else {
 			ret = hash_blob_stream(&stream,
 					       the_repository->hash_algo, oid,
odb/transaction.c +8 -4
index b16e07aebf..df4275151b 100644
@@ -2,14 +2,18 @@
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)
 {
+	int ret;
+
 	if (odb->transaction)
-		return NULL;
+		return -1;
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	ret = odb_source_begin_transaction(odb->sources, out);
+	odb->transaction = *out;
 
-	return odb->transaction;
+	return ret;
 }
 
 void odb_transaction_commit(struct odb_transaction *transaction)
odb/transaction.h +16 -4
index d52f0533ce..36032a5365 100644
@@ -1,6 +1,8 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "git-compat-util.h"
+#include "gettext.h"
 #include "odb.h"
 #include "odb/source.h"
 
@@ -36,11 +38,21 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. Note that it is considered an error to start a new transaction if the
+ * ODB already has an inflight transaction pending.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}
 
 /*
  * Commits an ODB transaction making the written objects visible. If the
read-cache.c +5 -2
index 21ca58beea..d511d25834 100644
@@ -4012,6 +4012,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 		       const struct pathspec *pathspec, char *ps_matched,
 		       int include_sparse, int flags, int ignored_too )
 {
+	int inflight = !!repo->objects->transaction;
 	struct odb_transaction *transaction;
 	struct update_callback_data data;
 	struct rev_info rev;
@@ -4042,9 +4043,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	transaction = odb_transaction_begin(repo->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(repo->objects, &transaction);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 
 	release_revisions(&rev);
 	return !!data.add_errors;
Signature · 1 line
2.55.0.122.gf85a7e6620
#41 lore raw

Re: [PATCH v2 06/11] odb/transaction: propagate begin errors

Quoted 13 lines · Justin Tobler

When odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.

Quoted 4 lines

Note that odb_transaction_begin() now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.

Yeah, this change is very much welcome and results in much saner behaviour with less surprises. Thanks for making the change.

Quoted 17 lines
cache-tree.c +1 -1
index 184f7e2635..8eec1d4d52 100644
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,
 
 int cache_tree_update(struct index_state *istate, int flags)
 {
+	int inflight = !!the_repository->objects->transaction;
 	struct odb_transaction *transaction;
 	int skip, i;
 
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);

cache-tree.c int cache_tree_update(struct index_state *istate, int flags) · line 502

Quoted 10 lines
cache-tree.c +4 -1
+	if (!inflight)
+		odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 	trace2_region_leave("cache_tree", "update", istate->repo);
 	trace_performance_leave("cache_tree_update");
 	if (i < 0)

Callsites like this really make me wonder why we even care to create a transaction in the first place if we basically just commit it immediately anyway. And while it's a bit sad that we have so many sites where we don't really know whether we even have a transaction, I think it's a good change that we have now annotated them clearly. A subsequent patch series may then eventually refactor those sites so that we stop depending on odb->transaction and inject the transaction via a parameter.

odb/transaction.h line 4

Quoted 9 lines
odb/transaction.h +1 -0
index d52f0533ce..36032a5365 100644
@@ -1,6 +1,8 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "git-compat-util.h"

We typically don't include "git-compat-util.h" in header files.

Quoted 13 lines
odb/transaction.h +5 -3
@@ -36,11 +38,21 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. Note that it is considered an error to start a new transaction if the
+ * ODB already has an inflight transaction pending.
  */

odb/transaction.h struct odb_transaction { · line 55

Quoted 10 lines
odb/transaction.h +9 -1
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}

We could make it a bit simpler to use this function by continuing to return the transaction directly. But on the other hand this results in a more consistent interface.

Patrick

#42 lore raw

Re: [PATCH v2 06/11] odb/transaction: propagate begin errors

Quoted 17 lines · Patrick Steinhardt
Quoted 5 lines · Justin Tobler
cache-tree.c +0 -1
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);

cache-tree.c int cache_tree_update(struct index_state *istate, int flags) · line 502

Quoted 10 lines
cache-tree.c +4 -1
+	if (!inflight)
+		odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 	trace2_region_leave("cache_tree", "update", istate->repo);
 	trace_performance_leave("cache_tree_update");
 	if (i < 0)
Quoted 8 lines

Callsites like this really make me wonder why we even care to create a transaction in the first place if we basically just commit it immediately anyway. And while it's a bit sad that we have so many sites where we don't really know whether we even have a transaction, I think it's a good change that we have now annotated them clearly. A subsequent patch series may then eventually refactor those sites so that we stop depending on odb->transaction and inject the transaction via a parameter.

Call sites like the one mentioned above are using ODB transactions as an optimization to batch the full fsyncs in bulk. In cases where the is not already a transaction, they start one to take advantage of it.

I fully agree though that an ODB transaction should ideally be started at a higher layer and wired down to these call sites. I have a couple of patches in my tree that start to tackle this which I plan to send in another series. :)

Quoted 11 lines

odb/transaction.h line 4

Quoted 9 lines
odb/transaction.h +1 -0
index d52f0533ce..36032a5365 100644
@@ -1,6 +1,8 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "git-compat-util.h"

We typically don't include "git-compat-util.h" in header files.

Will remove. Thanks

Quoted 19 lines
Quoted 19 lines
odb/transaction.h +10 -4
@@ -36,11 +38,21 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. Note that it is considered an error to start a new transaction if the
+ * ODB already has an inflight transaction pending.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
Quoted 8 lines

odb/transaction.h struct odb_transaction { · line 55

Quoted 4 lines
odb/transaction.h +4 -0
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}

We could make it a bit simpler to use this function by continuing to return the transaction directly. But on the other hand this results in a more consistent interface.

Ya, I was a bit back and forth about this myself. I ultimately landed on keeping a more consistent interface though. Happy to change if others feel differently though.

-Justin

#43 lore raw

Re: [PATCH v2 06/11] odb/transaction: propagate begin errors

Quoted 27 lines · Justin Tobler
Quoted 17 lines · Patrick Steinhardt
Quoted 5 lines · Justin Tobler
cache-tree.c +0 -1
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);

cache-tree.c int cache_tree_update(struct index_state *istate, int flags) · line 502

Quoted 10 lines
cache-tree.c +4 -1
+	if (!inflight)
+		odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 	trace2_region_leave("cache_tree", "update", istate->repo);
 	trace_performance_leave("cache_tree_update");
 	if (i < 0)
Quoted 8 lines

Callsites like this really make me wonder why we even care to create a transaction in the first place if we basically just commit it immediately anyway. And while it's a bit sad that we have so many sites where we don't really know whether we even have a transaction, I think it's a good change that we have now annotated them clearly. A subsequent patch series may then eventually refactor those sites so that we stop depending on odb->transaction and inject the transaction via a parameter.

Quoted 3 lines

Call sites like the one mentioned above are using ODB transactions as an optimization to batch the full fsyncs in bulk. In cases where the is not already a transaction, they start one to take advantage of it.

I know. But in the case where we don't want to batch we create the transaction anyway as far as I can see, and then we commit it immediately. So arguably, we could've just odb_write_object() and call it a day.

Quoted 4 lines

I fully agree though that an ODB transaction should ideally be started at a higher layer and wired down to these call sites. I have a couple of patches in my tree that start to tackle this which I plan to send in another series. :)

Yeah, let's not worry about that too much for now then. One step at a time :)

Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
odb/transaction.h +10 -4
@@ -36,11 +38,21 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. Note that it is considered an error to start a new transaction if the
+ * ODB already has an inflight transaction pending.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
Quoted 8 lines

odb/transaction.h struct odb_transaction { · line 55

Quoted 4 lines
odb/transaction.h +4 -0
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}

We could make it a bit simpler to use this function by continuing to return the transaction directly. But on the other hand this results in a more consistent interface.

Quoted 3 lines

Ya, I was a bit back and forth about this myself. I ultimately landed on keeping a more consistent interface though. Happy to change if others feel differently though.

As said, I can see both arguments. And ultimately, I don't care too much, so it's fine if this is just kept as-is.

Patrick

#44 lore raw

PATCH v2 7/11 odb/transaction: propagate commit errors

In reply to Justin Tobler

When odb_transaction_commit() is invoked, the return value of the backend commit callback is silently discarded. A backend has no way to signal that committing failed, such as when the "files" backend cannot migrate its temporary object directory into the permanent ODB.

In a subsequent commit, git-receive-pack(1) starts using ODB transaction to stage objects and consequently cares about such failures so it can handle the error appropriately. Change the commit callback signature to return an int error code and have odb_transaction_commit() forward it accordingly.

---
 odb/transaction.c | 13 ++++++++++---
 odb/transaction.h |  7 ++++---
 2 files changed, 14 insertions(+), 6 deletions(-)
odb/transaction.c +10 -3
index df4275151b..51af2c9a61 100644
@@ -16,19 +16,26 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
+	if (ret)
+		return ret;
+
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return 0;
 }
 
 int odb_transaction_write_object_stream(struct odb_transaction *transaction,
odb/transaction.h +4 -3
index 36032a5365..9557ee0fd2 100644
@@ -55,10 +55,11 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
 }
 
 /*
- * Commits an ODB transaction making the written objects visible. If the
- * specified transaction is NULL, the function is a no-op.
+ * Commits an ODB transaction making the written objects visible. Returns 0 on
+ * success, a negative error code otherwise. Note that, if the specified
+ * transaction is NULL, the function is a no-op and no error is returned.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);
 
 /*
  * Writes the object in the provided stream into the transaction. The resulting
Signature · 1 line
2.55.0.122.gf85a7e6620
#45 lore raw

Re: [PATCH v2 07/11] odb/transaction: propagate commit errors

Quoted 22 lines · Justin Tobler
odb/transaction.c +4 -2
index df4275151b..51af2c9a61 100644
@@ -16,19 +16,26 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 

odb/transaction.c int odb_transaction_begin(struct object_database *odb, · line 39

Quoted 10 lines
odb/transaction.c +6 -1
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
+	if (ret)
+		return ret;
+
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return 0;
 }

Doesn't this cause a leak now?

I think this interface here is doing the same mistake that our reference transactions did, where we automatically released the transaction on commit. That caused multiple lifetime issues with references all over the place.

This isn't an issue introduced by this patch series though, so it's fine to ignore this for now.

Patrick

#46 lore raw

Re: [PATCH v2 07/11] odb/transaction: propagate commit errors

Quoted 23 lines · Patrick Steinhardt
Quoted 22 lines · Justin Tobler
odb/transaction.c +4 -2
index df4275151b..51af2c9a61 100644
@@ -16,19 +16,26 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 
Quoted 12 lines

odb/transaction.c int odb_transaction_begin(struct object_database *odb, · line 39

Quoted 10 lines
odb/transaction.c +6 -1
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
+	if (ret)
+		return ret;
+
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return 0;
 }

Doesn't this cause a leak now?

Good call. Ya, if odb_transaction_commit() fails, we don't free the transaction. In the next version I'll go ahead and clear the transaction if we fail.

Quoted 4 lines

I think this interface here is doing the same mistake that our reference transactions did, where we automatically released the transaction on commit. That caused multiple lifetime issues with references all over the place.

Ya, it probaby makes sense to introduce a separate odb_transaction_release() function to make this explicit and update callers accordingly. I have another series I working on that introduces odb_transaction_abort(). This might be a good place to add it in too.

-Justin

#47 lore raw

PATCH v2 8/11 odb/transaction: add transaction env interface

In reply to Justin Tobler

The ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need access to uncommitted objects or write new objects in the staging area though.

Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction's staging area.

---
 object-file.c     | 14 ++++++++++++++
 odb/transaction.c |  8 ++++++++
 odb/transaction.h | 17 +++++++++++++++++
 3 files changed, 39 insertions(+)
object-file.c +14 -0
index cd1aa36462..9b8ee6f36c 100644
@@ -27,6 +27,7 @@
 #include "path.h"
 #include "read-cache-ll.h"
 #include "setup.h"
+#include "strvec.h"
 #include "tempfile.h"
 #include "tmp-objdir.h"
 
@@ -1687,6 +1688,18 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	return 0;
 }
 
+static int odb_transaction_files_env(struct odb_transaction *base,
+				     struct strvec *env)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+
+	odb_transaction_files_prepare(&transaction->base);
+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+
+	return 0;
+}
+
 int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out)
 {
@@ -1696,6 +1709,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	transaction->base.env = odb_transaction_files_env;
 	*out = &transaction->base;
 
 	return 0;
odb/transaction.c +8 -0
index 51af2c9a61..acb1c967e7 100644
@@ -44,3 +44,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 {
 	return transaction->write_object_stream(transaction, stream, len, oid);
 }
+
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)
+{
+	if (!transaction)
+		return 0;
+
+	return transaction->env(transaction, env);
+}
odb/transaction.h +17 -0
index 9557ee0fd2..1c6c97a53e 100644
@@ -35,6 +35,14 @@ struct odb_transaction {
 	int (*write_object_stream)(struct odb_transaction *transaction,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to populate the provided strvec with the
+	 * environment variables that a child process should inherit so that its
+	 * object writes participate in the transaction. Returns 0 on success, a
+	 * negative error code otherwise.
+	 */
+	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
 /*
@@ -70,4 +78,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 					struct odb_write_stream *stream,
 					size_t len, struct object_id *oid);
 
+/*
+ * Populates the provided strvec with the environment variables that a child
+ * process should inherit so that its object writes participate in the
+ * transaction, suitable for using via child_process.env. Returns 0 on success,
+ * a negative error code otherwise. Note that, if the specified transaction is
+ * NULL, the function is a no-op and no error is returned.
+ */
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env);
+
 #endif
Signature · 1 line
2.55.0.122.gf85a7e6620
#48 lore raw

PATCH v2 9/11 odb/transaction: introduce ODB transaction flags

In reply to Justin Tobler

The temporary directory used by git-receive-pack(1) to write objects is managed slightly differently than how it is done via ODB transactions:

- The temporary directory is eagerly created upfront, instead of waiting for the first object write.

- The prefix name of the temporary directory is "incoming" instead of

    "bulk-fsync".

In a subsequent commit, git-receive-pack(1) will use ODB transactions instead of tmp_objdir directly. To provide a means to configure the same transaction behavior, introduce enum odb_transaction_flags and the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB transactions using the "files" backend to be set up for git-receive-pack(1). Transaction call sites are updated accordingly to provide the required flag parameter.

---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 29 ++++++++++++++++++++++++++---
 object-file.h            |  4 +++-
 odb/source-files.c       |  5 +++--
 odb/source-inmemory.c    |  3 ++-
 odb/source-loose.c       |  3 ++-
 odb/source.h             |  9 ++++++---
 odb/transaction.c        |  5 +++--
 odb/transaction.h        | 15 +++++++++++----
 read-cache.c             |  2 +-
 13 files changed, 61 insertions(+), 22 deletions(-)
builtin/add.c +1 -1
index 3d5d9cfdb9..60ffbede2b 100644
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
builtin/unpack-objects.c +1 -1
index d0136cdd99..c3d0fc7507 100644
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
builtin/update-index.c +1 -1
index 17f3ea284c..bf6ea60ef4 100644
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
index 8eec1d4d52..99c6a0a7d0 100644
@@ -492,7 +492,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
 	if (!inflight)
-		odb_transaction_begin_or_die(the_repository->objects, &transaction);
+		odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	if (!inflight)
object-file.c +26 -3
index 9b8ee6f36c..d95bdabba5 100644
@@ -498,6 +498,7 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
+	const char *prefix;
 };
 
 static int odb_transaction_files_prepare(struct odb_transaction *base)
@@ -514,7 +515,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)
 	if (!transaction || transaction->objdir)
 		return 0;
 
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
+	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
 	if (!transaction->objdir)
 		return -1;
 
@@ -1359,7 +1360,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 			int inflight = !!transaction;
 
 			if (!inflight)
-				odb_transaction_begin_or_die(odb, &transaction);
+				odb_transaction_begin_or_die(odb, &transaction, 0);
 			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
@@ -1701,7 +1702,8 @@ static int odb_transaction_files_env(struct odb_transaction *base,
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1710,6 +1712,27 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different temporary directory
+		 * prefix.
+		 *
+		 * NEEDSWORK: This transaction flag is only used by the "files"
+		 * backend to special case temporary directory set up and
+		 * handling. Ideally transaction users should not have to care
+		 * though. To avoid this, we could eagerly create the temporary
+		 * directory and use the same prefix name for all transactions.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+
 	*out = &transaction->base;
 
 	return 0;
object-file.h +3 -1
index 1a023226ac..bdd2d67a2e 100644
@@ -5,6 +5,7 @@
 #include "object.h"
 #include "odb.h"
 #include "odb/source-loose.h"
+#include "odb/transaction.h"
 
 /* The maximum size for an object header. */
 #define MAX_HEADER_LEN 32
@@ -197,6 +198,7 @@ struct odb_transaction;
  * to make new objects visible.
  */
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out);
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags);
 
 #endif /* OBJECT_FILE_H */
odb/source-files.c +3 -2
index 2545bd81d4..534f48aad9 100644
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_files_begin_transaction(struct odb_source *source,
-					      struct odb_transaction **out)
+					      struct odb_transaction **out,
+					      enum odb_transaction_flags flags)
 {
-	return odb_transaction_files_begin(source, out);
+	return odb_transaction_files_begin(source, out, flags);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
odb/source-inmemory.c +2 -1
index e004566d76..9644d9d474 100644
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source,
 }
 
 static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED,
-						 struct odb_transaction **out UNUSED)
+						 struct odb_transaction **out UNUSED,
+						 enum odb_transaction_flags flags UNUSED)
 {
 	return error("in-memory source does not support transactions");
 }
odb/source-loose.c +2 -1
index 66e6bb8d3f..57c91986b4 100644
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED,
-					      struct odb_transaction **out UNUSED)
+					      struct odb_transaction **out UNUSED,
+					      enum odb_transaction_flags flags UNUSED)
 {
 	/* TODO: this is a known omission that we'll want to address eventually. */
 	return error("loose source does not support transactions");
odb/source.h +6 -3
index 2192a101b8..3790d03ff2 100644
@@ -3,6 +3,7 @@
 
 #include "object.h"
 #include "odb.h"
+#include "odb/transaction.h"
 
 enum odb_source_type {
 	/*
@@ -228,7 +229,8 @@ struct odb_source {
 	 * negative error code otherwise.
 	 */
 	int (*begin_transaction)(struct odb_source *source,
-				 struct odb_transaction **out);
+				 struct odb_transaction **out,
+				 enum odb_transaction_flags flags);
 
 	/*
 	 * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,
  * Returns 0 on success, a negative error code otherwise.
  */
 static inline int odb_source_begin_transaction(struct odb_source *source,
-					       struct odb_transaction **out)
+					       struct odb_transaction **out,
+					       enum odb_transaction_flags flags)
 {
-	return source->begin_transaction(source, out);
+	return source->begin_transaction(source, out, flags);
 }
 
 #endif
odb/transaction.c +3 -2
index acb1c967e7..007ab73c0c 100644
@@ -3,14 +3,15 @@
 #include "odb/transaction.h"
 
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out)
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags)
 {
 	int ret;
 
 	if (odb->transaction)
 		return -1;
 
-	ret = odb_source_begin_transaction(odb->sources, out);
+	ret = odb_source_begin_transaction(odb->sources, out, flags);
 	odb->transaction = *out;
 
 	return ret;
odb/transaction.h +11 -4
index 1c6c97a53e..b19f180aee 100644
@@ -4,7 +4,6 @@
 #include "git-compat-util.h"
 #include "gettext.h"
 #include "odb.h"
-#include "odb/source.h"
 
 /*
  * A transaction may be started for an object database prior to writing new
@@ -45,6 +44,12 @@ struct odb_transaction {
 	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
+/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
+	/* Configures the transaction for use with git-receive-pack(1). */
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()
@@ -53,12 +58,14 @@ struct odb_transaction {
  * ODB already has an inflight transaction pending.
  */
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out);
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags);
 
 static inline void odb_transaction_begin_or_die(struct object_database *odb,
-						struct odb_transaction **out)
+						struct odb_transaction **out,
+						enum odb_transaction_flags flags)
 {
-	if (odb_transaction_begin(odb, out))
+	if (odb_transaction_begin(odb, out, flags))
 		die(_("failed to start ODB transaction"));
 }
 
read-cache.c +1 -1
index d511d25834..50e2320c8d 100644
@@ -4044,7 +4044,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * may not have their own transaction active.
 	 */
 	if (!inflight)
-		odb_transaction_begin_or_die(repo->objects, &transaction);
+		odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	if (!inflight)
 		odb_transaction_commit(transaction);
Signature · 1 line
2.55.0.122.gf85a7e6620
#49 lore raw

Re: [PATCH v2 09/11] odb/transaction: introduce ODB transaction flags

Quoted 12 lines · Justin Tobler
object-file.c +4 -0
index 9b8ee6f36c..d95bdabba5 100644
@@ -1710,6 +1712,27 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*

object-file.c int odb_transaction_files_begin(struct odb_source *source, · line 1728

Quoted 10 lines
object-file.c +10 -0
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different temporary directory
+		 * prefix.
+		 *
+		 * NEEDSWORK: This transaction flag is only used by the "files"
+		 * backend to special case temporary directory set up and
+		 * handling. Ideally transaction users should not have to care
+		 * though. To avoid this, we could eagerly create the temporary
+		 * directory and use the same prefix name for all transactions.
+		 */

Yup, agreed, thanks for noting this here.

odb/transaction.h line 7

Quoted 9 lines
odb/transaction.h +0 -1
index 1c6c97a53e..b19f180aee 100644
@@ -4,7 +4,6 @@
 #include "git-compat-util.h"
 #include "gettext.h"
 #include "odb.h"
-#include "odb/source.h"

This is curious, and likely a result of you adding "odb/transaction.h" to "odb/source.h".

odb/transaction.h struct odb_transaction { · line 55

Quoted 13 lines
odb/transaction.h +6 -0
@@ -45,6 +44,12 @@ struct odb_transaction {
 	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
+/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
+	/* Configures the transaction for use with git-receive-pack(1). */
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()

And this is the reason you have to add the include, so that the flags are visible in both "odb/source.h" and in "odb/transaction.h".

This makes me wonder whether there's really much value in having this header here be split out of "odb/source.h".

Patrick

#50 lore raw

Re: [PATCH v2 09/11] odb/transaction: introduce ODB transaction flags

Quoted 14 lines · Patrick Steinhardt

odb/transaction.h struct odb_transaction { · line 55

Quoted 9 lines · Justin Tobler
odb/transaction.h +6 -0
+/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
+	/* Configures the transaction for use with git-receive-pack(1). */
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()

And this is the reason you have to add the include, so that the flags are visible in both "odb/source.h" and in "odb/transaction.h".

Quoted 2 lines

This makes me wonder whether there's really much value in having this header here be split out of "odb/source.h".

Ya, I've started wondering the same thing. A transaction implementation is always going to be tightly coupled to the ODB source it pertains too. It probably makes sense to merge "odb/transaction.{c,h}" with "odb/source.{c,h}". I'll leave it as-is for now and likely explore this is a future series though.

-Justin

#51 lore raw

PATCH v2 10/11 builtin/receive-pack: drop redundant tmpdir env

In reply to Justin Tobler

When performing the connectivity checks for a shallow ref in update_shallow_ref(), the child process environment variables are populated via tmp_objdir_env(). This is unnecessary though as update_shallow_ref() is only reached after tmp_objdir_migrate() has been performed which means there is no longer a temporary directory that needs to be shared with child processes.

Drop the call to tmp_objdir_env() accordingly.

---
 builtin/receive-pack.c | 1 -
 1 file changed, 1 deletion(-)
builtin/receive-pack.c +0 -1
index 19eb6a1b61..50bc05c70c 100644
@@ -1363,7 +1363,6 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
 		    !delayed_reachability_test(si, i))
 			oid_array_append(&extra, &si->shallow->oid[i]);
 
-	opt.env = tmp_objdir_env(tmp_objdir);
 	setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
 	if (check_connected(command_singleton_iterator, cmd, &opt)) {
 		rollback_shallow_file(the_repository, &shallow_lock);
Signature · 1 line
2.55.0.122.gf85a7e6620
#52 lore raw

Re: [PATCH v2 10/11] builtin/receive-pack: drop redundant tmpdir env

Quoted 6 lines · Justin Tobler

When performing the connectivity checks for a shallow ref in update_shallow_ref(), the child process environment variables are populated via tmp_objdir_env(). This is unnecessary though as update_shallow_ref() is only reached after tmp_objdir_migrate() has been performed which means there is no longer a temporary directory that needs to be shared with child processes.

Right. We call it transitively via either execute_commands_atomic() or execute_commands_not_atomic(), both of which are called after tmp_objdir_migrate().

Patrick

#53 lore raw

PATCH v2 11/11 builtin/receive-pack: stage incoming objects via ODB transactions

In reply to Justin Tobler

Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.

Note that the ODB transaction is now responsible for managing the primary and alternate ODBs for the repository. One small change as a result is that the temporary directory is now applied as the primary ODB in the main process instead of an alternate. This does not change anything for git-receive-pack(1) though because it only needs access to the newly written objects and doesn't care how exactly it is set up.

---
 builtin/receive-pack.c | 68 ++++++++++++++++++++++--------------------
 1 file changed, 35 insertions(+), 33 deletions(-)
builtin/receive-pack.c +35 -33
index 50bc05c70c..8b8c20dc1a 100644
@@ -37,7 +37,6 @@
 #include "sigchain.h"
 #include "string-list.h"
 #include "strvec.h"
-#include "tmp-objdir.h"
 #include "trace.h"
 #include "trace2.h"
 #include "version.h"
@@ -112,8 +111,6 @@ static enum {
 } use_keepalive;
 static int keepalive_in_sec = 5;
 
-static struct tmp_objdir *tmp_objdir;
-
 static struct proc_receive_ref {
 	unsigned int want_add:1,
 		     want_delete:1,
@@ -926,6 +923,7 @@ static void receive_hook_feed_state_free(void *data)
 static int run_receive_hook(struct command *commands,
 			    const char *hook_name,
 			    int skip_broken,
+			    struct odb_transaction *transaction,
 			    const struct string_list *push_options)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,
 		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
 	}
 
-	if (tmp_objdir)
-		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+	if (transaction)
+		odb_transaction_env(transaction, &opt.env);
 
 	prepare_push_cert_sha1(&opt);
 
@@ -1789,24 +1787,30 @@ static const struct object_id *command_singleton_iterator(void *cb_data)
 }
 
 static void set_connectivity_errors(struct command *commands,
-				    struct shallow_info *si)
+				    struct shallow_info *si,
+				    struct odb_transaction *transaction)
 {
 	struct command *cmd;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		struct command *singleton = cmd;
 		struct check_connected_options opt = CHECK_CONNECTED_INIT;
+		struct strvec env = STRVEC_INIT;
 
 		if (shallow_update && si->shallow_ref[cmd->index])
 			/* to be checked in update_shallow_ref() */
 			continue;
 
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
+
 		if (!check_connected(command_singleton_iterator, &singleton,
 				     &opt))
 			continue;
 
 		cmd->error_string = "missing necessary objects";
+
+		strvec_clear(&env);
 	}
 }
 
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,
 static void execute_commands(struct command *commands,
 			     const char *unpacker_error,
 			     struct shallow_info *si,
+			     struct odb_transaction *transaction,
 			     const struct string_list *push_options)
 {
 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
@@ -2043,6 +2048,8 @@ static void execute_commands(struct command *commands,
 	}
 
 	if (!skip_connectivity_check) {
+		struct strvec env = STRVEC_INIT;
+
 		if (use_sideband) {
 			memset(&muxer, 0, sizeof(muxer));
 			muxer.proc = copy_to_sideband;
@@ -2056,14 +2063,17 @@ static void execute_commands(struct command *commands,
 		data.si = si;
 		opt.err_fd = err_fd;
 		opt.progress = err_fd && !quiet;
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
 		opt.exclude_hidden_refs_section = "receive";
 
 		if (check_connected(iterate_receive_command_list, &data, &opt))
-			set_connectivity_errors(commands, si);
+			set_connectivity_errors(commands, si, transaction);
 
 		if (use_sideband)
 			finish_async(&muxer);
+
+		strvec_clear(&env);
 	}
 
 	reject_updates_to_hidden(commands);
@@ -2084,7 +2094,7 @@ static void execute_commands(struct command *commands,
 		}
 	}
 
-	if (run_receive_hook(commands, "pre-receive", 0, push_options)) {
+	if (run_receive_hook(commands, "pre-receive", 0, transaction, push_options)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "pre-receive hook declined";
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
+	if (odb_transaction_commit(transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;
 
 	check_aliased_updates(commands);
 
@@ -2325,7 +2334,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;
@@ -2350,20 +2360,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		strvec_push(&child.args, alt_shallow_file);
 	}
 
-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-	if (!tmp_objdir) {
-		if (err_fd > 0)
-			close(err_fd);
-		return "unable to create temporary object directory";
-	}
-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
-
-	/*
-	 * Normally we just pass the tmp_objdir environment to the child
-	 * processes that do the heavy lifting, but we may need to see these
-	 * objects ourselves to set up shallow information.
-	 */
-	tmp_objdir_add_as_alternate(tmp_objdir);
+	odb_transaction_env(transaction, &child.env);
 
 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
 		strvec_push(&child.args, "unpack-objects");
@@ -2430,13 +2427,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 	return NULL;
 }
 
-static const char *unpack_with_sideband(struct shallow_info *si)
+static const char *unpack_with_sideband(struct shallow_info *si,
+					struct odb_transaction *transaction)
 {
 	struct async muxer;
 	const char *ret;
 
 	if (!use_sideband)
-		return unpack(0, si);
+		return unpack(0, si, transaction);
 
 	use_keepalive = KEEPALIVE_AFTER_NUL;
 	memset(&muxer, 0, sizeof(muxer));
@@ -2445,7 +2443,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)
 	if (start_async(&muxer))
 		return NULL;
 
-	ret = unpack(muxer.in, si);
+	ret = unpack(muxer.in, si, transaction);
 
 	finish_async(&muxer);
 	return ret;
@@ -2622,6 +2620,7 @@ int cmd_receive_pack(int argc,
 	struct oid_array ref = OID_ARRAY_INIT;
 	struct shallow_info si;
 	struct packet_reader reader;
+	struct odb_transaction *transaction = NULL;
 
 	struct option options[] = {
 		OPT__QUIET(&quiet, N_("quiet")),
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
+				unpack_status = "unable to start object transaction";
+			else
+				unpack_status = unpack_with_sideband(&si, transaction);
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
-		execute_commands(commands, unpack_status, &si,
+		execute_commands(commands, unpack_status, &si, transaction,
 				 &push_options);
 		delete_tempfile(&pack_lockfile);
 		sigchain_push(SIGPIPE, SIG_IGN);
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,
 		else if (report_status)
 			report(commands, unpack_status);
 		sigchain_pop(SIGPIPE);
-		run_receive_hook(commands, "post-receive", 1,
+		run_receive_hook(commands, "post-receive", 1, NULL,
 				 &push_options);
 		run_update_post_hook(commands);
 		free_commands(commands);
Signature · 1 line
2.55.0.122.gf85a7e6620
#54 lore raw
In reply to Justin Tobler
Quoted 12 lines · Justin Tobler

Changes since V1:

- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling from odb_transaction_files_begin().
- odb_transaction_begin() now returns an error if there is already an inflight transaction pending instead of setting the out pointer to NULL.
- Updated odb_transaction_env() to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child

Quoted 8 lines

processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it from the_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.

I've got a couple smaller nits, but overall I'm quite happy with the shape of this series now. Thanks!

Patrick

#55 lore raw
Quoted 22 lines · Patrick Steinhardt
Quoted 12 lines · Justin Tobler

Changes since V1:

- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling from odb_transaction_files_begin().
- odb_transaction_begin() now returns an error if there is already an inflight transaction pending instead of setting the out pointer to NULL.
- Updated odb_transaction_env() to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child

Quoted 8 lines

processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it from the_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.

Quoted 2 lines

I've got a couple smaller nits, but overall I'm quite happy with the shape of this series now. Thanks!

Thanks for the review! I'll send another version later today. :)

-Justin

#56 lore raw

v3 cover receive-pack: use ODB transactions to stage object writes

In reply to Justin Tobler

Greetings,

This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.

Changes since V2:
- Clarified commit log reasoning for embedding flush_loose_object_transaction() logic in commit function.
- Started printed some error messages on transaction errors.
- Removed include statement.
- Fixed transaction leak on odb_transaction_commit() error.

Changes since V1:
- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling from odb_transaction_files_begin().
- odb_transaction_begin() now returns an error if there is already an inflight transaction pending instead of setting the out pointer to NULL.
- Updated odb_transaction_env() to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it from the_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.

Thanks,
-Justin

Justin Tobler (11):
object-file: rename files transaction prepare function
object-file: rename files transaction fsync function
object-file: embed transaction flush logic in commit function
object-file: drop check for inflight transactions
object-file: propagate files transaction errors odb/transaction: propagate begin errors odb/transaction: propagate commit errors odb/transaction: add transaction env interface odb/transaction: introduce ODB transaction flags builtin/receive-pack: drop redundant tmpdir env builtin/receive-pack: stage incoming objects via ODB transactions

 builtin/add.c            |   2 +-
 builtin/receive-pack.c   |  69 ++++++++---------
 builtin/unpack-objects.c |   2 +-
 builtin/update-index.c   |   2 +-
 cache-tree.c             |   7 +-
 object-file.c            | 159 +++++++++++++++++++++++++--------------
 object-file.h            |   8 +-
 odb/source-files.c       |   9 +--
 odb/source-inmemory.c    |   3 +-
 odb/source-loose.c       |   3 +-
 odb/source.h             |   9 ++-
 odb/transaction.c        |  32 ++++++--
 odb/transaction.h        |  59 ++++++++++++---
 read-cache.c             |   7 +-
 14 files changed, 241 insertions(+), 130 deletions(-)

Range-diff against v2:

 1:  9c14b219ad =  1:  9c14b219ad object-file: rename files transaction prepare function
 2:  5703a9e93b =  2:  5703a9e93b object-file: rename files transaction fsync function
 3:  4c37398ac8 !  3:  76204847f2 object-file: embed transaction flush logic in commit function

@@ Commit message
When a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into
- the primary. In a subsequent commit, the temporary directory is also
- used to write packfiles.
+ the primary and configuring the repository ODB source accordingly. The
+ function name here is a bit misleading because the helper is doing a bit
+ more than just "flushing" the transaction contents. Also, in a
+ subsequent commit, the transaction temporary directory is used to stage
+ packfiles and not just loose objects anymore.

- Instead of maintaining a separate helper function, embed the logic to
- flush and migrate the temporary directory directly into
- odb_transaction_files_commit().
+ Lift the helper function logic directly into
+ odb_transaction_files_commit() to more accurately signal to readers
+ the operation being performed.

 4:  623c6b02ea =  4:  c97eb7763f object-file: drop check for inflight transactions
 5:  ca59176657 !  5:  1f3a1f7714 object-file: propagate files transaction errors
    @@ object-file.c: static void odb_transaction_files_prepare(struct odb_transaction
     -	if (transaction->objdir)
     -		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
     +	if (!transaction->objdir)
    -+		return -1;
    ++		return error(_("unable to create temporary object directory"));
     +
     +	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
     +
    @@ object-file.c: static void odb_transaction_files_commit(struct odb_transaction *
      		 */
     -		tmp_objdir_migrate(transaction->objdir);
     +		if (tmp_objdir_migrate(transaction->objdir))
    -+			return -1;
    ++			return error(_("unable to migrate temporary objects"));
     +
      		transaction->objdir = NULL;
      	}
 6:  717a1ce9a7 !  6:  09d13272d5 odb/transaction: propagate begin errors
    @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
     
      ## odb/transaction.c ##
     @@
    + #include "git-compat-util.h"
    ++#include "gettext.h"
      #include "odb/source.h"
      #include "odb/transaction.h"
      
    @@ odb/transaction.c
     +
      	if (odb->transaction)
     -		return NULL;
    -+		return -1;
    ++		return error(_("object database transaction already pending"));
      
     -	odb_source_begin_transaction(odb->sources, &odb->transaction);
     +	ret = odb_source_begin_transaction(odb->sources, out);
    @@ odb/transaction.h
      #ifndef ODB_TRANSACTION_H
      #define ODB_TRANSACTION_H
      
    -+#include "git-compat-util.h"
     +#include "gettext.h"
      #include "odb.h"
      #include "odb/source.h"
 7:  ff8e133965 !  7:  12833d6773 odb/transaction: propagate commit errors
    @@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
      
     -	transaction->commit(transaction);
     +	ret = transaction->commit(transaction);
    -+	if (ret)
    -+		return ret;
    -+
      	transaction->source->odb->transaction = NULL;
      	free(transaction);
     +
    -+	return 0;
    ++	return ret;
      }
      
      int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 8:  264ba94b83 =  8:  f2586f2f34 odb/transaction: add transaction env interface
 9:  1e0a491ef2 !  9:  9d082b5e47 odb/transaction: introduce ODB transaction flags
    @@ object-file.c: static int odb_transaction_files_prepare(struct odb_transaction *
     -	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
     +	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
      	if (!transaction->objdir)
    - 		return -1;
    + 		return error(_("unable to create temporary object directory"));
      
     @@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
      			int inflight = !!transaction;
    @@ odb/transaction.c
      	int ret;
      
      	if (odb->transaction)
    - 		return -1;
    + 		return error(_("object database transaction already pending"));
      
     -	ret = odb_source_begin_transaction(odb->sources, out);
     +	ret = odb_source_begin_transaction(odb->sources, out, flags);
    @@ odb/transaction.c
     
      ## odb/transaction.h ##
     @@
    - #include "git-compat-util.h"
    + 
      #include "gettext.h"
      #include "odb.h"
     -#include "odb/source.h"
10:  6c8d878349 = 10:  e11d8a6676 builtin/receive-pack: drop redundant tmpdir env
11:  8db95fef56 = 11:  fee57c2817 builtin/receive-pack: stage incoming objects via ODB transactions

base-commit: ab776a62a78576513ee121424adb19597fbb7613

Signature · 1 line
2.55.0.122.gf85a7e6620
#57 lore raw

PATCH v3 1/11 object-file: rename files transaction prepare function

The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.

Rename the function to odb_transaction_files_prepare() accordingly.

---
 object-file.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
object-file.c +3 -3
index e3d92bbda2..a3eb8d71dd 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void prepare_loose_object_transaction(struct odb_transaction *base)
+static void odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf filename = STRBUF_INIT;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	odb_loose_path(loose, &filename, oid);
 
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,
 	int hdrlen;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
Signature · 1 line
2.55.0.122.gf85a7e6620
#58 lore raw

PATCH v3 2/11 object-file: rename files transaction fsync function

In reply to Justin Tobler

When writing an object to a "files" ODB transaction, a full hardware flush is not initially performed during the fsync in fsync_loose_object_transaction() and instead delayed until the transaction is later committed.

To be more consistent with other "files" ODB transaction helpers, rename the function to odb_transaction_files_fsync() accordingly. The conditional in the helper is also slightly restructured to improve clarity to readers.

---
 object-file.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
object-file.c +10 -6
index a3eb8d71dd..d68824bb44 100644
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
 }
 
-static void fsync_loose_object_transaction(struct odb_transaction *base,
-					   int fd, const char *filename)
+static void odb_transaction_files_fsync(struct odb_transaction *base,
+					int fd, const char *filename)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
 
+	if (!transaction || !transaction->objdir) {
+		fsync_or_die(fd, filename);
+		return;
+	}
+
 	/*
 	 * If we have an active ODB transaction, we issue a call that
 	 * cleans the filesystem page cache but avoids a hardware flush
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 	 * before renaming the objects to their final names as part of
 	 * flush_batch_fsync.
 	 */
-	if (!transaction || !transaction->objdir ||
-	    git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
+	if (git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
 		if (errno == ENOSYS)
 			warning(_("core.fsyncMethod = batch is unsupported on this platform"));
 		fsync_or_die(fd, filename);
@@ -553,7 +557,7 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
 	 * that all objects are durable before any renames occur. The code in
-	 * fsync_loose_object_transaction has already issued a writeout
+	 * odb_transaction_files_fsync has already issued a writeout
 	 * request, but it has not flushed any writeback cache in the storage
 	 * hardware or any filesystem logs. This fsync call acts as a barrier
 	 * to ensure that the data in each new object file is durable before
@@ -582,7 +586,7 @@ static void close_loose_object(struct odb_source_loose *loose,
 		goto out;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		fsync_loose_object_transaction(loose->base.odb->transaction, fd, filename);
+		odb_transaction_files_fsync(loose->base.odb->transaction, fd, filename);
 	else if (fsync_object_files > 0)
 		fsync_or_die(fd, filename);
 	else
Signature · 1 line
2.55.0.122.gf85a7e6620
#59 lore raw

PATCH v3 3/11 object-file: embed transaction flush logic in commit function

In reply to Justin Tobler

When a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary and configuring the repository ODB source accordingly. The function name here is a bit misleading because the helper is doing a bit more than just "flushing" the transaction contents. Also, in a subsequent commit, the transaction temporary directory is used to stage packfiles and not just loose objects anymore.

Lift the helper function logic directly into odb_transaction_files_commit() to more accurately signal to readers the operation being performed.

---
 object-file.c | 64 ++++++++++++++++++++++-----------------------------
 1 file changed, 28 insertions(+), 36 deletions(-)
object-file.c +28 -36
index d68824bb44..33bd6c6810 100644
@@ -543,41 +543,6 @@ static void odb_transaction_files_fsync(struct odb_transaction *base,
 	}
 }
 
-/*
- * Cleanup after batch-mode fsync_object_files.
- */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
-{
-	struct strbuf temp_path = STRBUF_INIT;
-	struct tempfile *temp;
-
-	if (!transaction->objdir)
-		return;
-
-	/*
-	 * Issue a full hardware flush against a temporary file to ensure
-	 * that all objects are durable before any renames occur. The code in
-	 * odb_transaction_files_fsync has already issued a writeout
-	 * request, but it has not flushed any writeback cache in the storage
-	 * hardware or any filesystem logs. This fsync call acts as a barrier
-	 * to ensure that the data in each new object file is durable before
-	 * the final name is visible.
-	 */
-	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
-		    repo_get_object_directory(transaction->base.source->odb->repo));
-	temp = xmks_tempfile(temp_path.buf);
-	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
-	delete_tempfile(&temp);
-	strbuf_release(&temp_path);
-
-	/*
-	 * Make the object files visible in the primary ODB after their data is
-	 * fully durable.
-	 */
-	tmp_objdir_migrate(transaction->objdir);
-	transaction->objdir = NULL;
-}
-
 /* Finalize a file on disk, and close it. */
 static void close_loose_object(struct odb_source_loose *loose,
 			       int fd, const char *filename)
@@ -1679,7 +1644,34 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (transaction->objdir) {
+		struct strbuf temp_path = STRBUF_INIT;
+		struct tempfile *temp;
+
+		/*
+		 * Issue a full hardware flush against a temporary file to ensure
+		 * that all objects are durable before any renames occur. The code in
+		 * odb_transaction_files_fsync has already issued a writeout
+		 * request, but it has not flushed any writeback cache in the storage
+		 * hardware or any filesystem logs. This fsync call acts as a barrier
+		 * to ensure that the data in each new object file is durable before
+		 * the final name is visible.
+		 */
+		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
+			    repo_get_object_directory(transaction->base.source->odb->repo));
+		temp = xmks_tempfile(temp_path.buf);
+		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
+		delete_tempfile(&temp);
+		strbuf_release(&temp_path);
+
+		/*
+		 * Make the object files visible in the primary ODB after their data is
+		 * fully durable.
+		 */
+		tmp_objdir_migrate(transaction->objdir);
+		transaction->objdir = NULL;
+	}
+
 	flush_packfile_transaction(transaction);
 }
 
Signature · 1 line
2.55.0.122.gf85a7e6620
#60 lore raw

Re: [PATCH v3 03/11] object-file: embed transaction flush logic in commit function

Quoted 9 lines · Justin Tobler

When a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary and configuring the repository ODB source accordingly. The function name here is a bit misleading because the helper is doing a bit more than just "flushing" the transaction contents. Also, in a subsequent commit, the transaction temporary directory is used to stage packfiles and not just loose objects anymore.

Quoted 3 lines

Lift the helper function logic directly into odb_transaction_files_commit() to more accurately signal to readers the operation being performed.

This line break makes my eyes bleed.

Patrick

#61 lore raw

PATCH v3 4/11 object-file: drop check for inflight transactions

In reply to Justin Tobler

ODB transactions are started via odb_transaction_begin() and contain validation to avoid starting multiple transactions at the same time. The "files" backend also has the same logic, but is redundant due to the generic layer already handling it. Drop this validation from the "files" backend accordingly.

---
 object-file.c | 4 ----
 object-file.h | 3 +--
 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -4
index 33bd6c6810..e51389833a 100644
@@ -1678,10 +1678,6 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 {
 	struct odb_transaction_files *transaction;
-	struct object_database *odb = source->odb;
-
-	if (odb->transaction)
-		return NULL;
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
object-file.h +1 -2
index 528c4e6e69..ea43d818f0 100644
@@ -194,8 +194,7 @@ struct odb_transaction;
 /*
  * Tell the object database to optimize for adding
  * multiple objects. odb_transaction_files_commit must be called
- * to make new objects visible. If a transaction is already
- * pending, NULL is returned.
+ * to make new objects visible.
  */
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
 
Signature · 1 line
2.55.0.122.gf85a7e6620
#62 lore raw

PATCH v3 5/11 object-file: propagate files transaction errors

In reply to Justin Tobler

The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle them as needed.

---
 object-file.c      | 26 ++++++++++++++++++--------
 object-file.h      |  3 ++-
 odb/source-files.c |  6 +-----
 odb/transaction.h  |  7 +++++--
 4 files changed, 26 insertions(+), 16 deletions(-)
object-file.c +18 -8
index e51389833a..3651605ea2 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return error(_("unable to create temporary object directory"));
+
+	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+
+	return 0;
 }
 
 static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ -1639,7 +1643,7 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 		 * Make the object files visible in the primary ODB after their data is
 		 * fully durable.
 		 */
-		tmp_objdir_migrate(transaction->objdir);
+		if (tmp_objdir_migrate(transaction->objdir))
+			return error(_("unable to migrate temporary objects"));
+
 		transaction->objdir = NULL;
 	}
 
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1683,6 +1692,7 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
-	return &transaction->base;
+	return 0;
 }
object-file.h +2 -1
index ea43d818f0..1a023226ac 100644
@@ -196,6 +196,7 @@ struct odb_transaction;
  * multiple objects. odb_transaction_files_commit must be called
  * to make new objects visible.
  */
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out);
 
 #endif /* OBJECT_FILE_H */
odb/source-files.c +1 -5
index 5bdd042922..2545bd81d4 100644
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 static int odb_source_files_begin_transaction(struct odb_source *source,
 					      struct odb_transaction **out)
 {
-	struct odb_transaction *tx = odb_transaction_files_begin(source);
-	if (!tx)
-		return -1;
-	*out = tx;
-	return 0;
+	return odb_transaction_files_begin(source, out);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
odb/transaction.h +5 -2
index 854fda06f5..d52f0533ce 100644
@@ -16,8 +16,11 @@ struct odb_transaction {
 	/* The ODB source the transaction is opened against. */
 	struct odb_source *source;
 
-	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	/*
+	 * The ODB source specific callback invoked to commit a transaction.
+	 * Returns 0 on success, a negative error code otherwise.
+	 */
+	int (*commit)(struct odb_transaction *transaction);
 
 	/*
 	 * This callback is expected to write the given object stream into
Signature · 1 line
2.55.0.122.gf85a7e6620
#63 lore raw

PATCH v3 6/11 odb/transaction: propagate begin errors

In reply to Justin Tobler

When odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.

Note that odb_transaction_begin() now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.

---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  7 +++++--
 object-file.c            | 10 +++++++---
 odb/transaction.c        | 13 +++++++++----
 odb/transaction.h        | 19 +++++++++++++++----
 read-cache.c             |  7 +++++--
 8 files changed, 44 insertions(+), 18 deletions(-)
builtin/add.c +1 -1
index c859f66519..3d5d9cfdb9 100644
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
builtin/unpack-objects.c +1 -1
index f3849bb654..d0136cdd99 100644
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
builtin/update-index.c +1 -1
index 3d6646c318..17f3ea284c 100644
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +5 -2
index 184f7e2635..8eec1d4d52 100644
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,
 
 int cache_tree_update(struct index_state *istate, int flags)
 {
+	int inflight = !!the_repository->objects->transaction;
 	struct odb_transaction *transaction;
 	int skip, i;
 
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 	trace2_region_leave("cache_tree", "update", istate->repo);
 	trace_performance_leave("cache_tree_update");
 	if (i < 0)
object-file.c +7 -3
index 3651605ea2..358684beae 100644
@@ -1354,13 +1354,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 
 		if (flags & INDEX_WRITE_OBJECT) {
 			struct object_database *odb = the_repository->objects;
-			struct odb_transaction *transaction = odb_transaction_begin(odb);
+			struct odb_transaction *transaction = odb->transaction;
+			int inflight = !!transaction;
 
-			ret = odb_transaction_write_object_stream(odb->transaction,
+			if (!inflight)
+				odb_transaction_begin_or_die(odb, &transaction);
+			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
 								  oid);
-			odb_transaction_commit(transaction);
+			if (!inflight)
+				odb_transaction_commit(transaction);
 		} else {
 			ret = hash_blob_stream(&stream,
 					       the_repository->hash_algo, oid,
odb/transaction.c +9 -4
index b16e07aebf..a5fba7f908 100644
@@ -1,15 +1,20 @@
 #include "git-compat-util.h"
+#include "gettext.h"
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)
 {
+	int ret;
+
 	if (odb->transaction)
-		return NULL;
+		return error(_("object database transaction already pending"));
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	ret = odb_source_begin_transaction(odb->sources, out);
+	odb->transaction = *out;
 
-	return odb->transaction;
+	return ret;
 }
 
 void odb_transaction_commit(struct odb_transaction *transaction)
odb/transaction.h +15 -4
index d52f0533ce..f5c43187c9 100644
@@ -1,6 +1,7 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "gettext.h"
 #include "odb.h"
 #include "odb/source.h"
 
@@ -36,11 +37,21 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. Note that it is considered an error to start a new transaction if the
+ * ODB already has an inflight transaction pending.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}
 
 /*
  * Commits an ODB transaction making the written objects visible. If the
read-cache.c +5 -2
index 21ca58beea..d511d25834 100644
@@ -4012,6 +4012,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 		       const struct pathspec *pathspec, char *ps_matched,
 		       int include_sparse, int flags, int ignored_too )
 {
+	int inflight = !!repo->objects->transaction;
 	struct odb_transaction *transaction;
 	struct update_callback_data data;
 	struct rev_info rev;
@@ -4042,9 +4043,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	transaction = odb_transaction_begin(repo->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(repo->objects, &transaction);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 
 	release_revisions(&rev);
 	return !!data.add_errors;
Signature · 1 line
2.55.0.122.gf85a7e6620
#64 lore raw

Re: [PATCH v3 06/11] odb/transaction: propagate begin errors

Quoted 34 lines · Justin Tobler

When odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.

Note that odb_transaction_begin() now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.

---
...

odb/transaction.c +3 -1
index b16e07aebf..a5fba7f908 100644
@@ -1,15 +1,20 @@
 #include "git-compat-util.h"
+#include "gettext.h"
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)

odb/transaction.c line 15

Quoted 10 lines
odb/transaction.c +5 -2
 {
+	int ret;
+
 	if (odb->transaction)
-		return NULL;
+		return error(_("object database transaction already pending"));
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	ret = odb_source_begin_transaction(odb->sources, out);
+	odb->transaction = *out;

Can odb_source_begin_transaction() ever fail? If so, and when it fails, would *out be left untouched? I am wondering if we want

	if (!(ret = odb_source_begin_transaction(odb->sources, out)))
        	odb->transaction = *out;

or something like that.

#65 lore raw

Re: [PATCH v3 06/11] odb/transaction: propagate begin errors

Quoted 43 lines · Junio C Hamano
Quoted 41 lines · Justin Tobler

When odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.

Note that odb_transaction_begin() now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.

---
...

odb/transaction.c +6 -2
index b16e07aebf..a5fba7f908 100644
@@ -1,15 +1,20 @@
 #include "git-compat-util.h"
+#include "gettext.h"
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)
 {
+	int ret;
+
 	if (odb->transaction)
-		return NULL;
+		return error(_("object database transaction already pending"));
 
Quoted 6 lines

odb/transaction.c line 15

Quoted 3 lines
odb/transaction.c +2 -1
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	ret = odb_source_begin_transaction(odb->sources, out);
+	odb->transaction = *out;

Can odb_source_begin_transaction() ever fail? If so, and when it fails, would *out be left untouched?

In this patch there it not yet a way for it to fail, but it can return an error later in the series. When an error is encountered though, *out _is_ left untouched.

Quoted 6 lines

I am wondering if we want

	if (!(ret = odb_source_begin_transaction(odb->sources, out)))
        	odb->transaction = *out;

or something like that.

I think it is a good idea to for odb_transaction_begin() to ensure the repository transaction is only set on success though. Will update in the next version. Thanks

-Justin

#66 lore raw

PATCH v3 7/11 odb/transaction: propagate commit errors

In reply to Justin Tobler

When odb_transaction_commit() is invoked, the return value of the backend commit callback is silently discarded. A backend has no way to signal that committing failed, such as when the "files" backend cannot migrate its temporary object directory into the permanent ODB.

In a subsequent commit, git-receive-pack(1) starts using ODB transaction to stage objects and consequently cares about such failures so it can handle the error appropriately. Change the commit callback signature to return an int error code and have odb_transaction_commit() forward it accordingly.

---
 odb/transaction.c | 10 +++++++---
 odb/transaction.h |  7 ++++---
 2 files changed, 11 insertions(+), 6 deletions(-)
odb/transaction.c +7 -3
index a5fba7f908..0a924e73f7 100644
@@ -17,19 +17,23 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return ret;
 }
 
 int odb_transaction_write_object_stream(struct odb_transaction *transaction,
odb/transaction.h +4 -3
index f5c43187c9..3b0a5a78e5 100644
@@ -54,10 +54,11 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
 }
 
 /*
- * Commits an ODB transaction making the written objects visible. If the
- * specified transaction is NULL, the function is a no-op.
+ * Commits an ODB transaction making the written objects visible. Returns 0 on
+ * success, a negative error code otherwise. Note that, if the specified
+ * transaction is NULL, the function is a no-op and no error is returned.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);
 
 /*
  * Writes the object in the provided stream into the transaction. The resulting
Signature · 1 line
2.55.0.122.gf85a7e6620
#67 lore raw

PATCH v3 8/11 odb/transaction: add transaction env interface

In reply to Justin Tobler

The ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need access to uncommitted objects or write new objects in the staging area though.

Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction's staging area.

---
 object-file.c     | 14 ++++++++++++++
 odb/transaction.c |  8 ++++++++
 odb/transaction.h | 17 +++++++++++++++++
 3 files changed, 39 insertions(+)
object-file.c +14 -0
index 358684beae..f0b066798a 100644
@@ -27,6 +27,7 @@
 #include "path.h"
 #include "read-cache-ll.h"
 #include "setup.h"
+#include "strvec.h"
 #include "tempfile.h"
 #include "tmp-objdir.h"
 
@@ -1687,6 +1688,18 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	return 0;
 }
 
+static int odb_transaction_files_env(struct odb_transaction *base,
+				     struct strvec *env)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+
+	odb_transaction_files_prepare(&transaction->base);
+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+
+	return 0;
+}
+
 int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out)
 {
@@ -1696,6 +1709,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	transaction->base.env = odb_transaction_files_env;
 	*out = &transaction->base;
 
 	return 0;
odb/transaction.c +8 -0
index 0a924e73f7..7f1b30945d 100644
@@ -42,3 +42,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 {
 	return transaction->write_object_stream(transaction, stream, len, oid);
 }
+
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)
+{
+	if (!transaction)
+		return 0;
+
+	return transaction->env(transaction, env);
+}
odb/transaction.h +17 -0
index 3b0a5a78e5..5e51ce5ca4 100644
@@ -34,6 +34,14 @@ struct odb_transaction {
 	int (*write_object_stream)(struct odb_transaction *transaction,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to populate the provided strvec with the
+	 * environment variables that a child process should inherit so that its
+	 * object writes participate in the transaction. Returns 0 on success, a
+	 * negative error code otherwise.
+	 */
+	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
 /*
@@ -69,4 +77,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 					struct odb_write_stream *stream,
 					size_t len, struct object_id *oid);
 
+/*
+ * Populates the provided strvec with the environment variables that a child
+ * process should inherit so that its object writes participate in the
+ * transaction, suitable for using via child_process.env. Returns 0 on success,
+ * a negative error code otherwise. Note that, if the specified transaction is
+ * NULL, the function is a no-op and no error is returned.
+ */
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env);
+
 #endif
Signature · 1 line
2.55.0.122.gf85a7e6620
#68 lore raw

Re: [PATCH v3 08/11] odb/transaction: add transaction env interface

object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1697

Quoted 7 lines · Justin Tobler
object-file.c +7 -0
+static int odb_transaction_files_env(struct odb_transaction *base,
+				     struct strvec *env)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+
+	odb_transaction_files_prepare(&transaction->base);

Can this fail? The caller of us would not notice that something went wrong, and ...

object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1698

Quoted 1 line
object-file.c +1 -0
+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));

... happily ends up using transaction->objdir that may not be appropriate for it to use if it fails, no?

Quoted 2 lines
(quoted diff) +2 -0
+	return 0;
+}
#69 lore raw

Re: [PATCH v3 08/11] odb/transaction: add transaction env interface

Quoted 13 lines · Junio C Hamano

object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1697

Quoted 7 lines · Justin Tobler
object-file.c +7 -0
+static int odb_transaction_files_env(struct odb_transaction *base,
+				     struct strvec *env)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+
+	odb_transaction_files_prepare(&transaction->base);

Can this fail? The caller of us would not notice that something went wrong, and ...

Quoted 4 lines

object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1698

Quoted 1 line
object-file.c +1 -0
+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));

... happily ends up using transaction->objdir that may not be appropriate for it to use if it fails, no?

Ya, odb_transaction_files_prepare() can fail here. In practice, failure results in no temporary directory being created which tmp_objdir_env() does handle gracefully, but we should ideally still be reported the failure back to callers. Will update in the next version.

-Justin

#70 lore raw

PATCH v3 9/11 odb/transaction: introduce ODB transaction flags

In reply to Justin Tobler

The temporary directory used by git-receive-pack(1) to write objects is managed slightly differently than how it is done via ODB transactions:

- The temporary directory is eagerly created upfront, instead of waiting for the first object write.

- The prefix name of the temporary directory is "incoming" instead of

    "bulk-fsync".

In a subsequent commit, git-receive-pack(1) will use ODB transactions instead of tmp_objdir directly. To provide a means to configure the same transaction behavior, introduce enum odb_transaction_flags and the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB transactions using the "files" backend to be set up for git-receive-pack(1). Transaction call sites are updated accordingly to provide the required flag parameter.

---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 29 ++++++++++++++++++++++++++---
 object-file.h            |  4 +++-
 odb/source-files.c       |  5 +++--
 odb/source-inmemory.c    |  3 ++-
 odb/source-loose.c       |  3 ++-
 odb/source.h             |  9 ++++++---
 odb/transaction.c        |  5 +++--
 odb/transaction.h        | 15 +++++++++++----
 read-cache.c             |  2 +-
 13 files changed, 61 insertions(+), 22 deletions(-)
builtin/add.c +1 -1
index 3d5d9cfdb9..60ffbede2b 100644
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
builtin/unpack-objects.c +1 -1
index d0136cdd99..c3d0fc7507 100644
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
builtin/update-index.c +1 -1
index 17f3ea284c..bf6ea60ef4 100644
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
index 8eec1d4d52..99c6a0a7d0 100644
@@ -492,7 +492,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
 	if (!inflight)
-		odb_transaction_begin_or_die(the_repository->objects, &transaction);
+		odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	if (!inflight)
object-file.c +26 -3
index f0b066798a..d2508d148f 100644
@@ -498,6 +498,7 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
+	const char *prefix;
 };
 
 static int odb_transaction_files_prepare(struct odb_transaction *base)
@@ -514,7 +515,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)
 	if (!transaction || transaction->objdir)
 		return 0;
 
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
+	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
 	if (!transaction->objdir)
 		return error(_("unable to create temporary object directory"));
 
@@ -1359,7 +1360,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 			int inflight = !!transaction;
 
 			if (!inflight)
-				odb_transaction_begin_or_die(odb, &transaction);
+				odb_transaction_begin_or_die(odb, &transaction, 0);
 			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
@@ -1701,7 +1702,8 @@ static int odb_transaction_files_env(struct odb_transaction *base,
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1710,6 +1712,27 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different temporary directory
+		 * prefix.
+		 *
+		 * NEEDSWORK: This transaction flag is only used by the "files"
+		 * backend to special case temporary directory set up and
+		 * handling. Ideally transaction users should not have to care
+		 * though. To avoid this, we could eagerly create the temporary
+		 * directory and use the same prefix name for all transactions.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+
 	*out = &transaction->base;
 
 	return 0;
object-file.h +3 -1
index 1a023226ac..bdd2d67a2e 100644
@@ -5,6 +5,7 @@
 #include "object.h"
 #include "odb.h"
 #include "odb/source-loose.h"
+#include "odb/transaction.h"
 
 /* The maximum size for an object header. */
 #define MAX_HEADER_LEN 32
@@ -197,6 +198,7 @@ struct odb_transaction;
  * to make new objects visible.
  */
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out);
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags);
 
 #endif /* OBJECT_FILE_H */
odb/source-files.c +3 -2
index 2545bd81d4..534f48aad9 100644
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_files_begin_transaction(struct odb_source *source,
-					      struct odb_transaction **out)
+					      struct odb_transaction **out,
+					      enum odb_transaction_flags flags)
 {
-	return odb_transaction_files_begin(source, out);
+	return odb_transaction_files_begin(source, out, flags);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
odb/source-inmemory.c +2 -1
index e004566d76..9644d9d474 100644
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source,
 }
 
 static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED,
-						 struct odb_transaction **out UNUSED)
+						 struct odb_transaction **out UNUSED,
+						 enum odb_transaction_flags flags UNUSED)
 {
 	return error("in-memory source does not support transactions");
 }
odb/source-loose.c +2 -1
index 66e6bb8d3f..57c91986b4 100644
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED,
-					      struct odb_transaction **out UNUSED)
+					      struct odb_transaction **out UNUSED,
+					      enum odb_transaction_flags flags UNUSED)
 {
 	/* TODO: this is a known omission that we'll want to address eventually. */
 	return error("loose source does not support transactions");
odb/source.h +6 -3
index 2192a101b8..3790d03ff2 100644
@@ -3,6 +3,7 @@
 
 #include "object.h"
 #include "odb.h"
+#include "odb/transaction.h"
 
 enum odb_source_type {
 	/*
@@ -228,7 +229,8 @@ struct odb_source {
 	 * negative error code otherwise.
 	 */
 	int (*begin_transaction)(struct odb_source *source,
-				 struct odb_transaction **out);
+				 struct odb_transaction **out,
+				 enum odb_transaction_flags flags);
 
 	/*
 	 * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,
  * Returns 0 on success, a negative error code otherwise.
  */
 static inline int odb_source_begin_transaction(struct odb_source *source,
-					       struct odb_transaction **out)
+					       struct odb_transaction **out,
+					       enum odb_transaction_flags flags)
 {
-	return source->begin_transaction(source, out);
+	return source->begin_transaction(source, out, flags);
 }
 
 #endif
odb/transaction.c +3 -2
index 7f1b30945d..edf5488c81 100644
@@ -4,14 +4,15 @@
 #include "odb/transaction.h"
 
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out)
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags)
 {
 	int ret;
 
 	if (odb->transaction)
 		return error(_("object database transaction already pending"));
 
-	ret = odb_source_begin_transaction(odb->sources, out);
+	ret = odb_source_begin_transaction(odb->sources, out, flags);
 	odb->transaction = *out;
 
 	return ret;
odb/transaction.h +11 -4
index 5e51ce5ca4..4cb2eafcbf 100644
@@ -3,7 +3,6 @@
 
 #include "gettext.h"
 #include "odb.h"
-#include "odb/source.h"
 
 /*
  * A transaction may be started for an object database prior to writing new
@@ -44,6 +43,12 @@ struct odb_transaction {
 	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
+/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
+	/* Configures the transaction for use with git-receive-pack(1). */
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()
@@ -52,12 +57,14 @@ struct odb_transaction {
  * ODB already has an inflight transaction pending.
  */
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out);
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags);
 
 static inline void odb_transaction_begin_or_die(struct object_database *odb,
-						struct odb_transaction **out)
+						struct odb_transaction **out,
+						enum odb_transaction_flags flags)
 {
-	if (odb_transaction_begin(odb, out))
+	if (odb_transaction_begin(odb, out, flags))
 		die(_("failed to start ODB transaction"));
 }
 
read-cache.c +1 -1
index d511d25834..50e2320c8d 100644
@@ -4044,7 +4044,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * may not have their own transaction active.
 	 */
 	if (!inflight)
-		odb_transaction_begin_or_die(repo->objects, &transaction);
+		odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	if (!inflight)
 		odb_transaction_commit(transaction);
Signature · 1 line
2.55.0.122.gf85a7e6620
#71 lore raw

PATCH v3 10/11 builtin/receive-pack: drop redundant tmpdir env

In reply to Justin Tobler

When performing the connectivity checks for a shallow ref in update_shallow_ref(), the child process environment variables are populated via tmp_objdir_env(). This is unnecessary though as update_shallow_ref() is only reached after tmp_objdir_migrate() has been performed which means there is no longer a temporary directory that needs to be shared with child processes.

Drop the call to tmp_objdir_env() accordingly.

---
 builtin/receive-pack.c | 1 -
 1 file changed, 1 deletion(-)
builtin/receive-pack.c +0 -1
index 19eb6a1b61..50bc05c70c 100644
@@ -1363,7 +1363,6 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
 		    !delayed_reachability_test(si, i))
 			oid_array_append(&extra, &si->shallow->oid[i]);
 
-	opt.env = tmp_objdir_env(tmp_objdir);
 	setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
 	if (check_connected(command_singleton_iterator, cmd, &opt)) {
 		rollback_shallow_file(the_repository, &shallow_lock);
Signature · 1 line
2.55.0.122.gf85a7e6620
#72 lore raw

PATCH v3 11/11 builtin/receive-pack: stage incoming objects via ODB transactions

In reply to Justin Tobler

Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.

Note that the ODB transaction is now responsible for managing the primary and alternate ODBs for the repository. One small change as a result is that the temporary directory is now applied as the primary ODB in the main process instead of an alternate. This does not change anything for git-receive-pack(1) though because it only needs access to the newly written objects and doesn't care how exactly it is set up.

---
 builtin/receive-pack.c | 68 ++++++++++++++++++++++--------------------
 1 file changed, 35 insertions(+), 33 deletions(-)
builtin/receive-pack.c +35 -33
index 50bc05c70c..8b8c20dc1a 100644
@@ -37,7 +37,6 @@
 #include "sigchain.h"
 #include "string-list.h"
 #include "strvec.h"
-#include "tmp-objdir.h"
 #include "trace.h"
 #include "trace2.h"
 #include "version.h"
@@ -112,8 +111,6 @@ static enum {
 } use_keepalive;
 static int keepalive_in_sec = 5;
 
-static struct tmp_objdir *tmp_objdir;
-
 static struct proc_receive_ref {
 	unsigned int want_add:1,
 		     want_delete:1,
@@ -926,6 +923,7 @@ static void receive_hook_feed_state_free(void *data)
 static int run_receive_hook(struct command *commands,
 			    const char *hook_name,
 			    int skip_broken,
+			    struct odb_transaction *transaction,
 			    const struct string_list *push_options)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,
 		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
 	}
 
-	if (tmp_objdir)
-		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+	if (transaction)
+		odb_transaction_env(transaction, &opt.env);
 
 	prepare_push_cert_sha1(&opt);
 
@@ -1789,24 +1787,30 @@ static const struct object_id *command_singleton_iterator(void *cb_data)
 }
 
 static void set_connectivity_errors(struct command *commands,
-				    struct shallow_info *si)
+				    struct shallow_info *si,
+				    struct odb_transaction *transaction)
 {
 	struct command *cmd;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		struct command *singleton = cmd;
 		struct check_connected_options opt = CHECK_CONNECTED_INIT;
+		struct strvec env = STRVEC_INIT;
 
 		if (shallow_update && si->shallow_ref[cmd->index])
 			/* to be checked in update_shallow_ref() */
 			continue;
 
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
+
 		if (!check_connected(command_singleton_iterator, &singleton,
 				     &opt))
 			continue;
 
 		cmd->error_string = "missing necessary objects";
+
+		strvec_clear(&env);
 	}
 }
 
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,
 static void execute_commands(struct command *commands,
 			     const char *unpacker_error,
 			     struct shallow_info *si,
+			     struct odb_transaction *transaction,
 			     const struct string_list *push_options)
 {
 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
@@ -2043,6 +2048,8 @@ static void execute_commands(struct command *commands,
 	}
 
 	if (!skip_connectivity_check) {
+		struct strvec env = STRVEC_INIT;
+
 		if (use_sideband) {
 			memset(&muxer, 0, sizeof(muxer));
 			muxer.proc = copy_to_sideband;
@@ -2056,14 +2063,17 @@ static void execute_commands(struct command *commands,
 		data.si = si;
 		opt.err_fd = err_fd;
 		opt.progress = err_fd && !quiet;
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
 		opt.exclude_hidden_refs_section = "receive";
 
 		if (check_connected(iterate_receive_command_list, &data, &opt))
-			set_connectivity_errors(commands, si);
+			set_connectivity_errors(commands, si, transaction);
 
 		if (use_sideband)
 			finish_async(&muxer);
+
+		strvec_clear(&env);
 	}
 
 	reject_updates_to_hidden(commands);
@@ -2084,7 +2094,7 @@ static void execute_commands(struct command *commands,
 		}
 	}
 
-	if (run_receive_hook(commands, "pre-receive", 0, push_options)) {
+	if (run_receive_hook(commands, "pre-receive", 0, transaction, push_options)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "pre-receive hook declined";
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
+	if (odb_transaction_commit(transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;
 
 	check_aliased_updates(commands);
 
@@ -2325,7 +2334,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;
@@ -2350,20 +2360,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		strvec_push(&child.args, alt_shallow_file);
 	}
 
-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-	if (!tmp_objdir) {
-		if (err_fd > 0)
-			close(err_fd);
-		return "unable to create temporary object directory";
-	}
-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
-
-	/*
-	 * Normally we just pass the tmp_objdir environment to the child
-	 * processes that do the heavy lifting, but we may need to see these
-	 * objects ourselves to set up shallow information.
-	 */
-	tmp_objdir_add_as_alternate(tmp_objdir);
+	odb_transaction_env(transaction, &child.env);
 
 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
 		strvec_push(&child.args, "unpack-objects");
@@ -2430,13 +2427,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 	return NULL;
 }
 
-static const char *unpack_with_sideband(struct shallow_info *si)
+static const char *unpack_with_sideband(struct shallow_info *si,
+					struct odb_transaction *transaction)
 {
 	struct async muxer;
 	const char *ret;
 
 	if (!use_sideband)
-		return unpack(0, si);
+		return unpack(0, si, transaction);
 
 	use_keepalive = KEEPALIVE_AFTER_NUL;
 	memset(&muxer, 0, sizeof(muxer));
@@ -2445,7 +2443,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)
 	if (start_async(&muxer))
 		return NULL;
 
-	ret = unpack(muxer.in, si);
+	ret = unpack(muxer.in, si, transaction);
 
 	finish_async(&muxer);
 	return ret;
@@ -2622,6 +2620,7 @@ int cmd_receive_pack(int argc,
 	struct oid_array ref = OID_ARRAY_INIT;
 	struct shallow_info si;
 	struct packet_reader reader;
+	struct odb_transaction *transaction = NULL;
 
 	struct option options[] = {
 		OPT__QUIET(&quiet, N_("quiet")),
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
+				unpack_status = "unable to start object transaction";
+			else
+				unpack_status = unpack_with_sideband(&si, transaction);
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
-		execute_commands(commands, unpack_status, &si,
+		execute_commands(commands, unpack_status, &si, transaction,
 				 &push_options);
 		delete_tempfile(&pack_lockfile);
 		sigchain_push(SIGPIPE, SIG_IGN);
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,
 		else if (report_status)
 			report(commands, unpack_status);
 		sigchain_pop(SIGPIPE);
-		run_receive_hook(commands, "post-receive", 1,
+		run_receive_hook(commands, "post-receive", 1, NULL,
 				 &push_options);
 		run_update_post_hook(commands);
 		free_commands(commands);
Signature · 1 line
2.55.0.122.gf85a7e6620
#73 lore raw

Re: [PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions

builtin/receive-pack.c static void execute_commands_atomic(struct command *commands, · line 2037

Quoted 9 lines · Justin Tobler
builtin/receive-pack.c +1 -0
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,
 static void execute_commands(struct command *commands,
 			     const char *unpacker_error,
 			     struct shallow_info *si,
+			     struct odb_transaction *transaction,
 			     const struct string_list *push_options)
 {
 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
...

Hidden in the context early in this function is an error return. When unpacker_error string is non NULL, we mark all the commands in the linked commands list as failed, and return early from this function.

Quoted 5 lines
builtin/receive-pack.c +0 -1
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {

builtin/receive-pack.c static void execute_commands(struct command *commands, · line 2126

Quoted 10 lines
builtin/receive-pack.c +1 -1
+	if (odb_transaction_commit(transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;
 
 	check_aliased_updates(commands);

In the "happy case", execute_commands() would commit the transaction before going on to do the execute_commands_{atomic,nonatomic}() that appears later in it.

builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2708

Quoted 6 lines
builtin/receive-pack.c +1 -1
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))

In the "main" program, we start a transaction here, and

builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2711

Quoted 3 lines
builtin/receive-pack.c +3 -0
+				unpack_status = "unable to start object transaction";
+			else
+				unpack_status = unpack_with_sideband(&si, transaction);

then call unpack_with_sideband(). It may fail.

builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2716

Quoted 6 lines
builtin/receive-pack.c +1 -1
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
-		execute_commands(commands, unpack_status, &si,
+		execute_commands(commands, unpack_status, &si, transaction,
 				 &push_options);

And in such a case, execute_commands() returns without committing the transaction. Is there a need to add and make an odb_transaction_abort() call or something in such a case? Everything should be cleaned up upon process exit, and on file based backends, we probably let the tempfile/lockfile API do their thing to clean up, but are there other things we may want to clean up?

Quoted 11 lines
builtin/receive-pack.c +1 -1
 		delete_tempfile(&pack_lockfile);
 		sigchain_push(SIGPIPE, SIG_IGN);
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,
 		else if (report_status)
 			report(commands, unpack_status);
 		sigchain_pop(SIGPIPE);
-		run_receive_hook(commands, "post-receive", 1,
+		run_receive_hook(commands, "post-receive", 1, NULL,
 				 &push_options);
 		run_update_post_hook(commands);
 		free_commands(commands);
#74 lore raw

Re: [PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions

Quoted 14 lines · Junio C Hamano

builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2716

Quoted 6 lines · Justin Tobler
builtin/receive-pack.c +1 -1
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
-		execute_commands(commands, unpack_status, &si,
+		execute_commands(commands, unpack_status, &si, transaction,
 				 &push_options);

And in such a case, execute_commands() returns without committing the transaction. Is there a need to add and make an odb_transaction_abort() call or something in such a case? Everything should be cleaned up upon process exit, and on file based backends, we probably let the tempfile/lockfile API do their thing to clean up, but are there other things we may want to clean up?

As you mentioned, if we exit before committing the ODB transaction, the temporary directory will get cleaned up when the process exits. I don't think there is anything else we need to cleanup that wouldn't be handled at exit though. Regardless, I do plan to add odb_transaction_abort() in a followup series and I think it would be nice to have an explicit "abort" here when we know that we are not going to commit anyways. I would like to defer this to my next series though.

-Justin

#75 lore raw

Re: [PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions

Quoted 16 lines · Justin Tobler
Quoted 14 lines · Junio C Hamano

builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2716

Quoted 6 lines · Justin Tobler
builtin/receive-pack.c +1 -1
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
-		execute_commands(commands, unpack_status, &si,
+		execute_commands(commands, unpack_status, &si, transaction,
 				 &push_options);

And in such a case, execute_commands() returns without committing the transaction. Is there a need to add and make an odb_transaction_abort() call or something in such a case? Everything should be cleaned up upon process exit, and on file based backends, we probably let the tempfile/lockfile API do their thing to clean up, but are there other things we may want to clean up?

Quoted 7 lines

As you mentioned, if we exit before committing the ODB transaction, the temporary directory will get cleaned up when the process exits. I don't think there is anything else we need to cleanup that wouldn't be handled at exit though. Regardless, I do plan to add odb_transaction_abort() in a followup series and I think it would be nice to have an explicit "abort" here when we know that we are not going to commit anyways. I would like to defer this to my next series though.

Sounds good. We cannot trigger receive-pack as a subroutine call in a long running daemon until that happens, but that is OK for now. One step at a time.

#76 lore raw
In reply to Justin Tobler
Quoted 6 lines · Justin Tobler

Changes since V2:
- Clarified commit log reasoning for embedding flush_loose_object_transaction() logic in commit function.
- Started printed some error messages on transaction errors.
- Removed include statement.
- Fixed transaction leak on odb_transaction_commit() error.

Thanks, the changes all look good to me and I don't have anything else to add.

Patrick

#77 lore raw

v4 cover receive-pack: use ODB transactions to stage object writes

In reply to Justin Tobler

Greetings,

This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.

Changes since V3:
- Removed ugly line break in commit message to prevent eye strain.
- odb_transaction_begin() now only sets the repository transaction on success.
- odb_transaction_env() now bubbles up error when failing to create the temporary directory.

Changes since V2:
- Clarified commit log reasoning for embedding flush_loose_object_transaction() logic in commit function.
- Started printed some error messages on transaction errors.
- Removed include statement.
- Fixed transaction leak on odb_transaction_commit() error.

Changes since V1:
- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling from odb_transaction_files_begin().
- odb_transaction_begin() now returns an error if there is already an inflight transaction pending instead of setting the out pointer to NULL.
- Updated odb_transaction_env() to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it from the_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.

Thanks,
-Justin

Justin Tobler (11):
object-file: rename files transaction prepare function
object-file: rename files transaction fsync function
object-file: embed transaction flush logic in commit function
object-file: drop check for inflight transactions
object-file: propagate files transaction errors odb/transaction: propagate begin errors odb/transaction: propagate commit errors odb/transaction: add transaction env interface odb/transaction: introduce ODB transaction flags builtin/receive-pack: drop redundant tmpdir env builtin/receive-pack: stage incoming objects via ODB transactions

 builtin/add.c            |   2 +-
 builtin/receive-pack.c   |  69 ++++++++---------
 builtin/unpack-objects.c |   2 +-
 builtin/update-index.c   |   2 +-
 cache-tree.c             |   7 +-
 object-file.c            | 161 +++++++++++++++++++++++++--------------
 object-file.h            |   8 +-
 odb/source-files.c       |   9 +--
 odb/source-inmemory.c    |   3 +-
 odb/source-loose.c       |   3 +-
 odb/source.h             |   9 ++-
 odb/transaction.c        |  33 ++++++--
 odb/transaction.h        |  59 +++++++++++---
 read-cache.c             |   7 +-
 14 files changed, 244 insertions(+), 130 deletions(-)

Range-diff against v3:

 1:  9c14b219ad =  1:  9c14b219ad object-file: rename files transaction prepare function
 2:  5703a9e93b =  2:  5703a9e93b object-file: rename files transaction fsync function
 3:  76204847f2 !  3:  70267741b0 object-file: embed transaction flush logic in commit function

@@ Commit message
subsequent commit, the transaction temporary directory is used to stage packfiles and not just loose objects anymore.

- Lift the helper function logic directly into
- odb_transaction_files_commit() to more accurately signal to readers
- the operation being performed.
+ Lift the helper function logic into odb_transaction_files_commit() to
+ more accurately signal to readers the operation being performed.

 4:  c97eb7763f =  4:  34cd3822c5 object-file: drop check for inflight transactions
 5:  1f3a1f7714 =  5:  240aa3475f object-file: propagate files transaction errors
 6:  09d13272d5 !  6:  0d91310fac odb/transaction: propagate begin errors
    @@ odb/transaction.c
      
     -	odb_source_begin_transaction(odb->sources, &odb->transaction);
     +	ret = odb_source_begin_transaction(odb->sources, out);
    -+	odb->transaction = *out;
    ++	if (!ret)
    ++		odb->transaction = *out;
      
     -	return odb->transaction;
     +	return ret;
 7:  12833d6773 =  7:  5e4680ed75 odb/transaction: propagate commit errors
 8:  f2586f2f34 !  8:  babcf6b156 odb/transaction: add transaction env interface
    @@ object-file.c: static int odb_transaction_files_commit(struct odb_transaction *b
     +{
     +	struct odb_transaction_files *transaction =
     +		container_of(base, struct odb_transaction_files, base);
    ++	int ret;
     +
    -+	odb_transaction_files_prepare(&transaction->base);
    -+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
    ++	ret = odb_transaction_files_prepare(&transaction->base);
    ++	if (!ret)
    ++		strvec_pushv(env, tmp_objdir_env(transaction->objdir));
     +
    -+	return 0;
    ++	return ret;
     +}
     +
      int odb_transaction_files_begin(struct odb_source *source,

9: 9d082b5e47 ! 9: 96f2a21eec odb/transaction: introduce ODB transaction flags
@@ odb/transaction.c

- ret = odb_source_begin_transaction(odb->sources, out);
+ ret = odb_source_begin_transaction(odb->sources, out, flags);
- odb->transaction = *out;
+ if (!ret)
+ odb->transaction = *out;

- return ret;

## odb/transaction.h ##
@@

10:  e11d8a6676 = 10:  56718f1190 builtin/receive-pack: drop redundant tmpdir env
11:  fee57c2817 = 11:  5197a19fbf builtin/receive-pack: stage incoming objects via ODB transactions

base-commit: ab776a62a78576513ee121424adb19597fbb7613

Signature · 1 line
2.55.0.122.gf85a7e6620
#78 lore raw

PATCH v4 1/11 object-file: rename files transaction prepare function

The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.

Rename the function to odb_transaction_files_prepare() accordingly.

---
 object-file.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
object-file.c +3 -3
index e3d92bbda2..a3eb8d71dd 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void prepare_loose_object_transaction(struct odb_transaction *base)
+static void odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,
 	static struct strbuf filename = STRBUF_INIT;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	odb_loose_path(loose, &filename, oid);
 
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,
 	int hdrlen;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		prepare_loose_object_transaction(loose->base.odb->transaction);
+		odb_transaction_files_prepare(loose->base.odb->transaction);
 
 	/* Since oid is not determined, save tmp file to odb path. */
 	strbuf_addf(&filename, "%s/", loose->base.path);
Signature · 1 line
2.55.0.122.gf85a7e6620
#79 lore raw

PATCH v4 2/11 object-file: rename files transaction fsync function

In reply to Justin Tobler

When writing an object to a "files" ODB transaction, a full hardware flush is not initially performed during the fsync in fsync_loose_object_transaction() and instead delayed until the transaction is later committed.

To be more consistent with other "files" ODB transaction helpers, rename the function to odb_transaction_files_fsync() accordingly. The conditional in the helper is also slightly restructured to improve clarity to readers.

---
 object-file.c | 16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
object-file.c +10 -6
index a3eb8d71dd..d68824bb44 100644
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
 }
 
-static void fsync_loose_object_transaction(struct odb_transaction *base,
-					   int fd, const char *filename)
+static void odb_transaction_files_fsync(struct odb_transaction *base,
+					int fd, const char *filename)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
 
+	if (!transaction || !transaction->objdir) {
+		fsync_or_die(fd, filename);
+		return;
+	}
+
 	/*
 	 * If we have an active ODB transaction, we issue a call that
 	 * cleans the filesystem page cache but avoids a hardware flush
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
 	 * before renaming the objects to their final names as part of
 	 * flush_batch_fsync.
 	 */
-	if (!transaction || !transaction->objdir ||
-	    git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
+	if (git_fsync(fd, FSYNC_WRITEOUT_ONLY) < 0) {
 		if (errno == ENOSYS)
 			warning(_("core.fsyncMethod = batch is unsupported on this platform"));
 		fsync_or_die(fd, filename);
@@ -553,7 +557,7 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
 	/*
 	 * Issue a full hardware flush against a temporary file to ensure
 	 * that all objects are durable before any renames occur. The code in
-	 * fsync_loose_object_transaction has already issued a writeout
+	 * odb_transaction_files_fsync has already issued a writeout
 	 * request, but it has not flushed any writeback cache in the storage
 	 * hardware or any filesystem logs. This fsync call acts as a barrier
 	 * to ensure that the data in each new object file is durable before
@@ -582,7 +586,7 @@ static void close_loose_object(struct odb_source_loose *loose,
 		goto out;
 
 	if (batch_fsync_enabled(FSYNC_COMPONENT_LOOSE_OBJECT))
-		fsync_loose_object_transaction(loose->base.odb->transaction, fd, filename);
+		odb_transaction_files_fsync(loose->base.odb->transaction, fd, filename);
 	else if (fsync_object_files > 0)
 		fsync_or_die(fd, filename);
 	else
Signature · 1 line
2.55.0.122.gf85a7e6620
#80 lore raw

PATCH v4 3/11 object-file: embed transaction flush logic in commit function

In reply to Justin Tobler

When a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary and configuring the repository ODB source accordingly. The function name here is a bit misleading because the helper is doing a bit more than just "flushing" the transaction contents. Also, in a subsequent commit, the transaction temporary directory is used to stage packfiles and not just loose objects anymore.

Lift the helper function logic into odb_transaction_files_commit() to more accurately signal to readers the operation being performed.

---
 object-file.c | 64 ++++++++++++++++++++++-----------------------------
 1 file changed, 28 insertions(+), 36 deletions(-)
object-file.c +28 -36
index d68824bb44..33bd6c6810 100644
@@ -543,41 +543,6 @@ static void odb_transaction_files_fsync(struct odb_transaction *base,
 	}
 }
 
-/*
- * Cleanup after batch-mode fsync_object_files.
- */
-static void flush_loose_object_transaction(struct odb_transaction_files *transaction)
-{
-	struct strbuf temp_path = STRBUF_INIT;
-	struct tempfile *temp;
-
-	if (!transaction->objdir)
-		return;
-
-	/*
-	 * Issue a full hardware flush against a temporary file to ensure
-	 * that all objects are durable before any renames occur. The code in
-	 * odb_transaction_files_fsync has already issued a writeout
-	 * request, but it has not flushed any writeback cache in the storage
-	 * hardware or any filesystem logs. This fsync call acts as a barrier
-	 * to ensure that the data in each new object file is durable before
-	 * the final name is visible.
-	 */
-	strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
-		    repo_get_object_directory(transaction->base.source->odb->repo));
-	temp = xmks_tempfile(temp_path.buf);
-	fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
-	delete_tempfile(&temp);
-	strbuf_release(&temp_path);
-
-	/*
-	 * Make the object files visible in the primary ODB after their data is
-	 * fully durable.
-	 */
-	tmp_objdir_migrate(transaction->objdir);
-	transaction->objdir = NULL;
-}
-
 /* Finalize a file on disk, and close it. */
 static void close_loose_object(struct odb_source_loose *loose,
 			       int fd, const char *filename)
@@ -1679,7 +1644,34 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
 
-	flush_loose_object_transaction(transaction);
+	if (transaction->objdir) {
+		struct strbuf temp_path = STRBUF_INIT;
+		struct tempfile *temp;
+
+		/*
+		 * Issue a full hardware flush against a temporary file to ensure
+		 * that all objects are durable before any renames occur. The code in
+		 * odb_transaction_files_fsync has already issued a writeout
+		 * request, but it has not flushed any writeback cache in the storage
+		 * hardware or any filesystem logs. This fsync call acts as a barrier
+		 * to ensure that the data in each new object file is durable before
+		 * the final name is visible.
+		 */
+		strbuf_addf(&temp_path, "%s/bulk_fsync_XXXXXX",
+			    repo_get_object_directory(transaction->base.source->odb->repo));
+		temp = xmks_tempfile(temp_path.buf);
+		fsync_or_die(get_tempfile_fd(temp), get_tempfile_path(temp));
+		delete_tempfile(&temp);
+		strbuf_release(&temp_path);
+
+		/*
+		 * Make the object files visible in the primary ODB after their data is
+		 * fully durable.
+		 */
+		tmp_objdir_migrate(transaction->objdir);
+		transaction->objdir = NULL;
+	}
+
 	flush_packfile_transaction(transaction);
 }
 
Signature · 1 line
2.55.0.122.gf85a7e6620
#81 lore raw

PATCH v4 4/11 object-file: drop check for inflight transactions

In reply to Justin Tobler

ODB transactions are started via odb_transaction_begin() and contain validation to avoid starting multiple transactions at the same time. The "files" backend also has the same logic, but is redundant due to the generic layer already handling it. Drop this validation from the "files" backend accordingly.

---
 object-file.c | 4 ----
 object-file.h | 3 +--
 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -4
index 33bd6c6810..e51389833a 100644
@@ -1678,10 +1678,6 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 {
 	struct odb_transaction_files *transaction;
-	struct object_database *odb = source->odb;
-
-	if (odb->transaction)
-		return NULL;
 
 	transaction = xcalloc(1, sizeof(*transaction));
 	transaction->base.source = source;
object-file.h +1 -2
index 528c4e6e69..ea43d818f0 100644
@@ -194,8 +194,7 @@ struct odb_transaction;
 /*
  * Tell the object database to optimize for adding
  * multiple objects. odb_transaction_files_commit must be called
- * to make new objects visible. If a transaction is already
- * pending, NULL is returned.
+ * to make new objects visible.
  */
 struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
 
Signature · 1 line
2.55.0.122.gf85a7e6620
#82 lore raw

PATCH v4 5/11 object-file: propagate files transaction errors

In reply to Justin Tobler

The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle them as needed.

---
 object-file.c      | 26 ++++++++++++++++++--------
 object-file.h      |  3 ++-
 odb/source-files.c |  6 +-----
 odb/transaction.h  |  7 +++++--
 4 files changed, 26 insertions(+), 16 deletions(-)
object-file.c +18 -8
index e51389833a..3651605ea2 100644
@@ -499,7 +499,7 @@ struct odb_transaction_files {
 	struct transaction_packfile packfile;
 };
 
-static void odb_transaction_files_prepare(struct odb_transaction *base)
+static int odb_transaction_files_prepare(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)
 	 * added at the time they call odb_transaction_files_begin.
 	 */
 	if (!transaction || transaction->objdir)
-		return;
+		return 0;
 
 	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
-	if (transaction->objdir)
-		tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+	if (!transaction->objdir)
+		return error(_("unable to create temporary object directory"));
+
+	tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+
+	return 0;
 }
 
 static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ -1639,7 +1643,7 @@ int read_loose_object(struct repository *repo,
 	return ret;
 }
 
-static void odb_transaction_files_commit(struct odb_transaction *base)
+static int odb_transaction_files_commit(struct odb_transaction *base)
 {
 	struct odb_transaction_files *transaction =
 		container_of(base, struct odb_transaction_files, base);
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)
 		 * Make the object files visible in the primary ODB after their data is
 		 * fully durable.
 		 */
-		tmp_objdir_migrate(transaction->objdir);
+		if (tmp_objdir_migrate(transaction->objdir))
+			return error(_("unable to migrate temporary objects"));
+
 		transaction->objdir = NULL;
 	}
 
 	flush_packfile_transaction(transaction);
+
+	return 0;
 }
 
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1683,6 +1692,7 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	*out = &transaction->base;
 
-	return &transaction->base;
+	return 0;
 }
object-file.h +2 -1
index ea43d818f0..1a023226ac 100644
@@ -196,6 +196,7 @@ struct odb_transaction;
  * multiple objects. odb_transaction_files_commit must be called
  * to make new objects visible.
  */
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source);
+int odb_transaction_files_begin(struct odb_source *source,
+				struct odb_transaction **out);
 
 #endif /* OBJECT_FILE_H */
odb/source-files.c +1 -5
index 5bdd042922..2545bd81d4 100644
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 static int odb_source_files_begin_transaction(struct odb_source *source,
 					      struct odb_transaction **out)
 {
-	struct odb_transaction *tx = odb_transaction_files_begin(source);
-	if (!tx)
-		return -1;
-	*out = tx;
-	return 0;
+	return odb_transaction_files_begin(source, out);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
odb/transaction.h +5 -2
index 854fda06f5..d52f0533ce 100644
@@ -16,8 +16,11 @@ struct odb_transaction {
 	/* The ODB source the transaction is opened against. */
 	struct odb_source *source;
 
-	/* The ODB source specific callback invoked to commit a transaction. */
-	void (*commit)(struct odb_transaction *transaction);
+	/*
+	 * The ODB source specific callback invoked to commit a transaction.
+	 * Returns 0 on success, a negative error code otherwise.
+	 */
+	int (*commit)(struct odb_transaction *transaction);
 
 	/*
 	 * This callback is expected to write the given object stream into
Signature · 1 line
2.55.0.122.gf85a7e6620
#83 lore raw

PATCH v4 6/11 odb/transaction: propagate begin errors

In reply to Justin Tobler

When odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.

In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.

Note that odb_transaction_begin() now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.

---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  7 +++++--
 object-file.c            | 10 +++++++---
 odb/transaction.c        | 14 ++++++++++----
 odb/transaction.h        | 19 +++++++++++++++----
 read-cache.c             |  7 +++++--
 8 files changed, 45 insertions(+), 18 deletions(-)
builtin/add.c +1 -1
index c859f66519..3d5d9cfdb9 100644
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	transaction = odb_transaction_begin(repo->objects);
+	odb_transaction_begin_or_die(repo->objects, &transaction);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
builtin/unpack-objects.c +1 -1
index f3849bb654..d0136cdd99 100644
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
builtin/update-index.c +1 -1
index 3d6646c318..17f3ea284c 100644
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	transaction = odb_transaction_begin(the_repository->objects);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +5 -2
index 184f7e2635..8eec1d4d52 100644
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,
 
 int cache_tree_update(struct index_state *istate, int flags)
 {
+	int inflight = !!the_repository->objects->transaction;
 	struct odb_transaction *transaction;
 	int skip, i;
 
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)
 
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
-	transaction = odb_transaction_begin(the_repository->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(the_repository->objects, &transaction);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 	trace2_region_leave("cache_tree", "update", istate->repo);
 	trace_performance_leave("cache_tree_update");
 	if (i < 0)
object-file.c +7 -3
index 3651605ea2..358684beae 100644
@@ -1354,13 +1354,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 
 		if (flags & INDEX_WRITE_OBJECT) {
 			struct object_database *odb = the_repository->objects;
-			struct odb_transaction *transaction = odb_transaction_begin(odb);
+			struct odb_transaction *transaction = odb->transaction;
+			int inflight = !!transaction;
 
-			ret = odb_transaction_write_object_stream(odb->transaction,
+			if (!inflight)
+				odb_transaction_begin_or_die(odb, &transaction);
+			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
 								  oid);
-			odb_transaction_commit(transaction);
+			if (!inflight)
+				odb_transaction_commit(transaction);
 		} else {
 			ret = hash_blob_stream(&stream,
 					       the_repository->hash_algo, oid,
odb/transaction.c +10 -4
index b16e07aebf..b6da4a3942 100644
@@ -1,15 +1,21 @@
 #include "git-compat-util.h"
+#include "gettext.h"
 #include "odb/source.h"
 #include "odb/transaction.h"
 
-struct odb_transaction *odb_transaction_begin(struct object_database *odb)
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out)
 {
+	int ret;
+
 	if (odb->transaction)
-		return NULL;
+		return error(_("object database transaction already pending"));
 
-	odb_source_begin_transaction(odb->sources, &odb->transaction);
+	ret = odb_source_begin_transaction(odb->sources, out);
+	if (!ret)
+		odb->transaction = *out;
 
-	return odb->transaction;
+	return ret;
 }
 
 void odb_transaction_commit(struct odb_transaction *transaction)
odb/transaction.h +15 -4
index d52f0533ce..f5c43187c9 100644
@@ -1,6 +1,7 @@
 #ifndef ODB_TRANSACTION_H
 #define ODB_TRANSACTION_H
 
+#include "gettext.h"
 #include "odb.h"
 #include "odb/source.h"
 
@@ -36,11 +37,21 @@ struct odb_transaction {
 };
 
 /*
- * Starts an ODB transaction. Subsequent objects are written to the transaction
- * and not committed until odb_transaction_commit() is invoked on the
- * transaction. If the ODB already has a pending transaction, NULL is returned.
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
+ * error. Note that it is considered an error to start a new transaction if the
+ * ODB already has an inflight transaction pending.
  */
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
+			  struct odb_transaction **out);
+
+static inline void odb_transaction_begin_or_die(struct object_database *odb,
+						struct odb_transaction **out)
+{
+	if (odb_transaction_begin(odb, out))
+		die(_("failed to start ODB transaction"));
+}
 
 /*
  * Commits an ODB transaction making the written objects visible. If the
read-cache.c +5 -2
index 21ca58beea..d511d25834 100644
@@ -4012,6 +4012,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 		       const struct pathspec *pathspec, char *ps_matched,
 		       int include_sparse, int flags, int ignored_too )
 {
+	int inflight = !!repo->objects->transaction;
 	struct odb_transaction *transaction;
 	struct update_callback_data data;
 	struct rev_info rev;
@@ -4042,9 +4043,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * This function is invoked from commands other than 'add', which
 	 * may not have their own transaction active.
 	 */
-	transaction = odb_transaction_begin(repo->objects);
+	if (!inflight)
+		odb_transaction_begin_or_die(repo->objects, &transaction);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
-	odb_transaction_commit(transaction);
+	if (!inflight)
+		odb_transaction_commit(transaction);
 
 	release_revisions(&rev);
 	return !!data.add_errors;
Signature · 1 line
2.55.0.122.gf85a7e6620
#84 lore raw

PATCH v4 7/11 odb/transaction: propagate commit errors

In reply to Justin Tobler

When odb_transaction_commit() is invoked, the return value of the backend commit callback is silently discarded. A backend has no way to signal that committing failed, such as when the "files" backend cannot migrate its temporary object directory into the permanent ODB.

In a subsequent commit, git-receive-pack(1) starts using ODB transaction to stage objects and consequently cares about such failures so it can handle the error appropriately. Change the commit callback signature to return an int error code and have odb_transaction_commit() forward it accordingly.

---
 odb/transaction.c | 10 +++++++---
 odb/transaction.h |  7 ++++---
 2 files changed, 11 insertions(+), 6 deletions(-)
odb/transaction.c +7 -3
index b6da4a3942..249ef4d9b7 100644
@@ -18,19 +18,23 @@ int odb_transaction_begin(struct object_database *odb,
 	return ret;
 }
 
-void odb_transaction_commit(struct odb_transaction *transaction)
+int odb_transaction_commit(struct odb_transaction *transaction)
 {
+	int ret;
+
 	if (!transaction)
-		return;
+		return 0;
 
 	/*
 	 * Ensure the transaction ending matches the pending transaction.
 	 */
 	ASSERT(transaction == transaction->source->odb->transaction);
 
-	transaction->commit(transaction);
+	ret = transaction->commit(transaction);
 	transaction->source->odb->transaction = NULL;
 	free(transaction);
+
+	return ret;
 }
 
 int odb_transaction_write_object_stream(struct odb_transaction *transaction,
odb/transaction.h +4 -3
index f5c43187c9..3b0a5a78e5 100644
@@ -54,10 +54,11 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,
 }
 
 /*
- * Commits an ODB transaction making the written objects visible. If the
- * specified transaction is NULL, the function is a no-op.
+ * Commits an ODB transaction making the written objects visible. Returns 0 on
+ * success, a negative error code otherwise. Note that, if the specified
+ * transaction is NULL, the function is a no-op and no error is returned.
  */
-void odb_transaction_commit(struct odb_transaction *transaction);
+int odb_transaction_commit(struct odb_transaction *transaction);
 
 /*
  * Writes the object in the provided stream into the transaction. The resulting
Signature · 1 line
2.55.0.122.gf85a7e6620
#85 lore raw

PATCH v4 8/11 odb/transaction: add transaction env interface

In reply to Justin Tobler

The ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need access to uncommitted objects or write new objects in the staging area though.

Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction's staging area.

---
 object-file.c     | 16 ++++++++++++++++
 odb/transaction.c |  8 ++++++++
 odb/transaction.h | 17 +++++++++++++++++
 3 files changed, 41 insertions(+)
object-file.c +16 -0
index 358684beae..39b92e275c 100644
@@ -27,6 +27,7 @@
 #include "path.h"
 #include "read-cache-ll.h"
 #include "setup.h"
+#include "strvec.h"
 #include "tempfile.h"
 #include "tmp-objdir.h"
 
@@ -1687,6 +1688,20 @@ static int odb_transaction_files_commit(struct odb_transaction *base)
 	return 0;
 }
 
+static int odb_transaction_files_env(struct odb_transaction *base,
+				     struct strvec *env)
+{
+	struct odb_transaction_files *transaction =
+		container_of(base, struct odb_transaction_files, base);
+	int ret;
+
+	ret = odb_transaction_files_prepare(&transaction->base);
+	if (!ret)
+		strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+
+	return ret;
+}
+
 int odb_transaction_files_begin(struct odb_source *source,
 				struct odb_transaction **out)
 {
@@ -1696,6 +1711,7 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.source = source;
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
+	transaction->base.env = odb_transaction_files_env;
 	*out = &transaction->base;
 
 	return 0;
odb/transaction.c +8 -0
index 249ef4d9b7..92ec8786a1 100644
@@ -43,3 +43,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 {
 	return transaction->write_object_stream(transaction, stream, len, oid);
 }
+
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)
+{
+	if (!transaction)
+		return 0;
+
+	return transaction->env(transaction, env);
+}
odb/transaction.h +17 -0
index 3b0a5a78e5..5e51ce5ca4 100644
@@ -34,6 +34,14 @@ struct odb_transaction {
 	int (*write_object_stream)(struct odb_transaction *transaction,
 				   struct odb_write_stream *stream, size_t len,
 				   struct object_id *oid);
+
+	/*
+	 * This callback is expected to populate the provided strvec with the
+	 * environment variables that a child process should inherit so that its
+	 * object writes participate in the transaction. Returns 0 on success, a
+	 * negative error code otherwise.
+	 */
+	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
 /*
@@ -69,4 +77,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,
 					struct odb_write_stream *stream,
 					size_t len, struct object_id *oid);
 
+/*
+ * Populates the provided strvec with the environment variables that a child
+ * process should inherit so that its object writes participate in the
+ * transaction, suitable for using via child_process.env. Returns 0 on success,
+ * a negative error code otherwise. Note that, if the specified transaction is
+ * NULL, the function is a no-op and no error is returned.
+ */
+int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env);
+
 #endif
Signature · 1 line
2.55.0.122.gf85a7e6620
#86 lore raw

PATCH v4 9/11 odb/transaction: introduce ODB transaction flags

In reply to Justin Tobler

The temporary directory used by git-receive-pack(1) to write objects is managed slightly differently than how it is done via ODB transactions:

- The temporary directory is eagerly created upfront, instead of waiting for the first object write.

- The prefix name of the temporary directory is "incoming" instead of

    "bulk-fsync".

In a subsequent commit, git-receive-pack(1) will use ODB transactions instead of tmp_objdir directly. To provide a means to configure the same transaction behavior, introduce enum odb_transaction_flags and the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB transactions using the "files" backend to be set up for git-receive-pack(1). Transaction call sites are updated accordingly to provide the required flag parameter.

---
 builtin/add.c            |  2 +-
 builtin/unpack-objects.c |  2 +-
 builtin/update-index.c   |  2 +-
 cache-tree.c             |  2 +-
 object-file.c            | 29 ++++++++++++++++++++++++++---
 object-file.h            |  4 +++-
 odb/source-files.c       |  5 +++--
 odb/source-inmemory.c    |  3 ++-
 odb/source-loose.c       |  3 ++-
 odb/source.h             |  9 ++++++---
 odb/transaction.c        |  5 +++--
 odb/transaction.h        | 15 +++++++++++----
 read-cache.c             |  2 +-
 13 files changed, 61 insertions(+), 22 deletions(-)
builtin/add.c +1 -1
index 3d5d9cfdb9..60ffbede2b 100644
@@ -581,7 +581,7 @@ int cmd_add(int argc,
 		string_list_clear(&only_match_skip_worktree, 0);
 	}
 
-	odb_transaction_begin_or_die(repo->objects, &transaction);
+	odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 
 	ps_matched = xcalloc(pathspec.nr, 1);
 	if (add_renormalize)
builtin/unpack-objects.c +1 -1
index d0136cdd99..c3d0fc7507 100644
@@ -598,7 +598,7 @@ static void unpack_all(void)
 		progress = start_progress(the_repository,
 					  _("Unpacking objects"), nr_objects);
 	CALLOC_ARRAY(obj_list, nr_objects);
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	for (i = 0; i < nr_objects; i++) {
 		unpack_one(i);
 		display_progress(progress, i + 1);
builtin/update-index.c +1 -1
index 17f3ea284c..bf6ea60ef4 100644
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,
 	 * Allow the object layer to optimize adding multiple objects in
 	 * a batch.
 	 */
-	odb_transaction_begin_or_die(the_repository->objects, &transaction);
+	odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	while (ctx.argc) {
 		if (parseopt_state != PARSE_OPT_DONE)
 			parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
index 8eec1d4d52..99c6a0a7d0 100644
@@ -492,7 +492,7 @@ int cache_tree_update(struct index_state *istate, int flags)
 	trace_performance_enter();
 	trace2_region_enter("cache_tree", "update", istate->repo);
 	if (!inflight)
-		odb_transaction_begin_or_die(the_repository->objects, &transaction);
+		odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
 	i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
 		       "", 0, &skip, flags);
 	if (!inflight)
object-file.c +26 -3
index 39b92e275c..0640a22009 100644
@@ -498,6 +498,7 @@ struct odb_transaction_files {
 
 	struct tmp_objdir *objdir;
 	struct transaction_packfile packfile;
+	const char *prefix;
 };
 
 static int odb_transaction_files_prepare(struct odb_transaction *base)
@@ -514,7 +515,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)
 	if (!transaction || transaction->objdir)
 		return 0;
 
-	transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
+	transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
 	if (!transaction->objdir)
 		return error(_("unable to create temporary object directory"));
 
@@ -1359,7 +1360,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,
 			int inflight = !!transaction;
 
 			if (!inflight)
-				odb_transaction_begin_or_die(odb, &transaction);
+				odb_transaction_begin_or_die(odb, &transaction, 0);
 			ret = odb_transaction_write_object_stream(transaction,
 								  &stream,
 								  xsize_t(st->st_size),
@@ -1703,7 +1704,8 @@ static int odb_transaction_files_env(struct odb_transaction *base,
 }
 
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out)
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags)
 {
 	struct odb_transaction_files *transaction;
 
@@ -1712,6 +1714,27 @@ int odb_transaction_files_begin(struct odb_source *source,
 	transaction->base.commit = odb_transaction_files_commit;
 	transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
 	transaction->base.env = odb_transaction_files_env;
+
+	transaction->prefix = "bulk-fsync";
+	if (flags & ODB_TRANSACTION_RECEIVE) {
+		/*
+		 * ODB transactions for git-receive-pack(1) eagerly create a
+		 * temporary directory and use a different temporary directory
+		 * prefix.
+		 *
+		 * NEEDSWORK: This transaction flag is only used by the "files"
+		 * backend to special case temporary directory set up and
+		 * handling. Ideally transaction users should not have to care
+		 * though. To avoid this, we could eagerly create the temporary
+		 * directory and use the same prefix name for all transactions.
+		 */
+		transaction->prefix = "incoming";
+		if (odb_transaction_files_prepare(&transaction->base)) {
+			free(transaction);
+			return -1;
+		}
+	}
+
 	*out = &transaction->base;
 
 	return 0;
object-file.h +3 -1
index 1a023226ac..bdd2d67a2e 100644
@@ -5,6 +5,7 @@
 #include "object.h"
 #include "odb.h"
 #include "odb/source-loose.h"
+#include "odb/transaction.h"
 
 /* The maximum size for an object header. */
 #define MAX_HEADER_LEN 32
@@ -197,6 +198,7 @@ struct odb_transaction;
  * to make new objects visible.
  */
 int odb_transaction_files_begin(struct odb_source *source,
-				struct odb_transaction **out);
+				struct odb_transaction **out,
+				enum odb_transaction_flags flags);
 
 #endif /* OBJECT_FILE_H */
odb/source-files.c +3 -2
index 2545bd81d4..534f48aad9 100644
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_files_begin_transaction(struct odb_source *source,
-					      struct odb_transaction **out)
+					      struct odb_transaction **out,
+					      enum odb_transaction_flags flags)
 {
-	return odb_transaction_files_begin(source, out);
+	return odb_transaction_files_begin(source, out, flags);
 }
 
 static int odb_source_files_read_alternates(struct odb_source *source,
odb/source-inmemory.c +2 -1
index e004566d76..9644d9d474 100644
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source,
 }
 
 static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED,
-						 struct odb_transaction **out UNUSED)
+						 struct odb_transaction **out UNUSED,
+						 enum odb_transaction_flags flags UNUSED)
 {
 	return error("in-memory source does not support transactions");
 }
odb/source-loose.c +2 -1
index 66e6bb8d3f..57c91986b4 100644
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source,
 }
 
 static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED,
-					      struct odb_transaction **out UNUSED)
+					      struct odb_transaction **out UNUSED,
+					      enum odb_transaction_flags flags UNUSED)
 {
 	/* TODO: this is a known omission that we'll want to address eventually. */
 	return error("loose source does not support transactions");
odb/source.h +6 -3
index 2192a101b8..3790d03ff2 100644
@@ -3,6 +3,7 @@
 
 #include "object.h"
 #include "odb.h"
+#include "odb/transaction.h"
 
 enum odb_source_type {
 	/*
@@ -228,7 +229,8 @@ struct odb_source {
 	 * negative error code otherwise.
 	 */
 	int (*begin_transaction)(struct odb_source *source,
-				 struct odb_transaction **out);
+				 struct odb_transaction **out,
+				 enum odb_transaction_flags flags);
 
 	/*
 	 * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,
  * Returns 0 on success, a negative error code otherwise.
  */
 static inline int odb_source_begin_transaction(struct odb_source *source,
-					       struct odb_transaction **out)
+					       struct odb_transaction **out,
+					       enum odb_transaction_flags flags)
 {
-	return source->begin_transaction(source, out);
+	return source->begin_transaction(source, out, flags);
 }
 
 #endif
odb/transaction.c +3 -2
index 92ec8786a1..dab7da6a9a 100644
@@ -4,14 +4,15 @@
 #include "odb/transaction.h"
 
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out)
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags)
 {
 	int ret;
 
 	if (odb->transaction)
 		return error(_("object database transaction already pending"));
 
-	ret = odb_source_begin_transaction(odb->sources, out);
+	ret = odb_source_begin_transaction(odb->sources, out, flags);
 	if (!ret)
 		odb->transaction = *out;
 
odb/transaction.h +11 -4
index 5e51ce5ca4..4cb2eafcbf 100644
@@ -3,7 +3,6 @@
 
 #include "gettext.h"
 #include "odb.h"
-#include "odb/source.h"
 
 /*
  * A transaction may be started for an object database prior to writing new
@@ -44,6 +43,12 @@ struct odb_transaction {
 	int (*env)(struct odb_transaction *transaction, struct strvec *env);
 };
 
+/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
+	/* Configures the transaction for use with git-receive-pack(1). */
+	ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
 /*
  * Starts an ODB transaction and returns it via `out`. Subsequent objects are
  * written to the transaction and not committed until odb_transaction_commit()
@@ -52,12 +57,14 @@ struct odb_transaction {
  * ODB already has an inflight transaction pending.
  */
 int odb_transaction_begin(struct object_database *odb,
-			  struct odb_transaction **out);
+			  struct odb_transaction **out,
+			  enum odb_transaction_flags flags);
 
 static inline void odb_transaction_begin_or_die(struct object_database *odb,
-						struct odb_transaction **out)
+						struct odb_transaction **out,
+						enum odb_transaction_flags flags)
 {
-	if (odb_transaction_begin(odb, out))
+	if (odb_transaction_begin(odb, out, flags))
 		die(_("failed to start ODB transaction"));
 }
 
read-cache.c +1 -1
index d511d25834..50e2320c8d 100644
@@ -4044,7 +4044,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,
 	 * may not have their own transaction active.
 	 */
 	if (!inflight)
-		odb_transaction_begin_or_die(repo->objects, &transaction);
+		odb_transaction_begin_or_die(repo->objects, &transaction, 0);
 	run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
 	if (!inflight)
 		odb_transaction_commit(transaction);
Signature · 1 line
2.55.0.122.gf85a7e6620
#87 lore raw

PATCH v4 10/11 builtin/receive-pack: drop redundant tmpdir env

In reply to Justin Tobler

When performing the connectivity checks for a shallow ref in update_shallow_ref(), the child process environment variables are populated via tmp_objdir_env(). This is unnecessary though as update_shallow_ref() is only reached after tmp_objdir_migrate() has been performed which means there is no longer a temporary directory that needs to be shared with child processes.

Drop the call to tmp_objdir_env() accordingly.

---
 builtin/receive-pack.c | 1 -
 1 file changed, 1 deletion(-)
builtin/receive-pack.c +0 -1
index 19eb6a1b61..50bc05c70c 100644
@@ -1363,7 +1363,6 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
 		    !delayed_reachability_test(si, i))
 			oid_array_append(&extra, &si->shallow->oid[i]);
 
-	opt.env = tmp_objdir_env(tmp_objdir);
 	setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
 	if (check_connected(command_singleton_iterator, cmd, &opt)) {
 		rollback_shallow_file(the_repository, &shallow_lock);
Signature · 1 line
2.55.0.122.gf85a7e6620
#88 lore raw

PATCH v4 11/11 builtin/receive-pack: stage incoming objects via ODB transactions

In reply to Justin Tobler

Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.

Note that the ODB transaction is now responsible for managing the primary and alternate ODBs for the repository. One small change as a result is that the temporary directory is now applied as the primary ODB in the main process instead of an alternate. This does not change anything for git-receive-pack(1) though because it only needs access to the newly written objects and doesn't care how exactly it is set up.

---
 builtin/receive-pack.c | 68 ++++++++++++++++++++++--------------------
 1 file changed, 35 insertions(+), 33 deletions(-)
builtin/receive-pack.c +35 -33
index 50bc05c70c..8b8c20dc1a 100644
@@ -37,7 +37,6 @@
 #include "sigchain.h"
 #include "string-list.h"
 #include "strvec.h"
-#include "tmp-objdir.h"
 #include "trace.h"
 #include "trace2.h"
 #include "version.h"
@@ -112,8 +111,6 @@ static enum {
 } use_keepalive;
 static int keepalive_in_sec = 5;
 
-static struct tmp_objdir *tmp_objdir;
-
 static struct proc_receive_ref {
 	unsigned int want_add:1,
 		     want_delete:1,
@@ -926,6 +923,7 @@ static void receive_hook_feed_state_free(void *data)
 static int run_receive_hook(struct command *commands,
 			    const char *hook_name,
 			    int skip_broken,
+			    struct odb_transaction *transaction,
 			    const struct string_list *push_options)
 {
 	struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,
 		strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
 	}
 
-	if (tmp_objdir)
-		strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
+	if (transaction)
+		odb_transaction_env(transaction, &opt.env);
 
 	prepare_push_cert_sha1(&opt);
 
@@ -1789,24 +1787,30 @@ static const struct object_id *command_singleton_iterator(void *cb_data)
 }
 
 static void set_connectivity_errors(struct command *commands,
-				    struct shallow_info *si)
+				    struct shallow_info *si,
+				    struct odb_transaction *transaction)
 {
 	struct command *cmd;
 
 	for (cmd = commands; cmd; cmd = cmd->next) {
 		struct command *singleton = cmd;
 		struct check_connected_options opt = CHECK_CONNECTED_INIT;
+		struct strvec env = STRVEC_INIT;
 
 		if (shallow_update && si->shallow_ref[cmd->index])
 			/* to be checked in update_shallow_ref() */
 			continue;
 
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
+
 		if (!check_connected(command_singleton_iterator, &singleton,
 				     &opt))
 			continue;
 
 		cmd->error_string = "missing necessary objects";
+
+		strvec_clear(&env);
 	}
 }
 
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,
 static void execute_commands(struct command *commands,
 			     const char *unpacker_error,
 			     struct shallow_info *si,
+			     struct odb_transaction *transaction,
 			     const struct string_list *push_options)
 {
 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
@@ -2043,6 +2048,8 @@ static void execute_commands(struct command *commands,
 	}
 
 	if (!skip_connectivity_check) {
+		struct strvec env = STRVEC_INIT;
+
 		if (use_sideband) {
 			memset(&muxer, 0, sizeof(muxer));
 			muxer.proc = copy_to_sideband;
@@ -2056,14 +2063,17 @@ static void execute_commands(struct command *commands,
 		data.si = si;
 		opt.err_fd = err_fd;
 		opt.progress = err_fd && !quiet;
-		opt.env = tmp_objdir_env(tmp_objdir);
+		odb_transaction_env(transaction, &env);
+		opt.env = env.v;
 		opt.exclude_hidden_refs_section = "receive";
 
 		if (check_connected(iterate_receive_command_list, &data, &opt))
-			set_connectivity_errors(commands, si);
+			set_connectivity_errors(commands, si, transaction);
 
 		if (use_sideband)
 			finish_async(&muxer);
+
+		strvec_clear(&env);
 	}
 
 	reject_updates_to_hidden(commands);
@@ -2084,7 +2094,7 @@ static void execute_commands(struct command *commands,
 		}
 	}
 
-	if (run_receive_hook(commands, "pre-receive", 0, push_options)) {
+	if (run_receive_hook(commands, "pre-receive", 0, transaction, push_options)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "pre-receive hook declined";
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,
 	 * Now we'll start writing out refs, which means the objects need
 	 * to be in their final positions so that other processes can see them.
 	 */
-	if (tmp_objdir_migrate(tmp_objdir) < 0) {
+	if (odb_transaction_commit(transaction)) {
 		for (cmd = commands; cmd; cmd = cmd->next) {
 			if (!cmd->error_string)
 				cmd->error_string = "unable to migrate objects to permanent storage";
 		}
 		return;
 	}
-	tmp_objdir = NULL;
 
 	check_aliased_updates(commands);
 
@@ -2325,7 +2334,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr)
 		     ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
 }
 
-static const char *unpack(int err_fd, struct shallow_info *si)
+static const char *unpack(int err_fd, struct shallow_info *si,
+			  struct odb_transaction *transaction)
 {
 	struct pack_header hdr;
 	const char *hdr_err;
@@ -2350,20 +2360,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 		strvec_push(&child.args, alt_shallow_file);
 	}
 
-	tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-	if (!tmp_objdir) {
-		if (err_fd > 0)
-			close(err_fd);
-		return "unable to create temporary object directory";
-	}
-	strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
-
-	/*
-	 * Normally we just pass the tmp_objdir environment to the child
-	 * processes that do the heavy lifting, but we may need to see these
-	 * objects ourselves to set up shallow information.
-	 */
-	tmp_objdir_add_as_alternate(tmp_objdir);
+	odb_transaction_env(transaction, &child.env);
 
 	if (ntohl(hdr.hdr_entries) < unpack_limit) {
 		strvec_push(&child.args, "unpack-objects");
@@ -2430,13 +2427,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)
 	return NULL;
 }
 
-static const char *unpack_with_sideband(struct shallow_info *si)
+static const char *unpack_with_sideband(struct shallow_info *si,
+					struct odb_transaction *transaction)
 {
 	struct async muxer;
 	const char *ret;
 
 	if (!use_sideband)
-		return unpack(0, si);
+		return unpack(0, si, transaction);
 
 	use_keepalive = KEEPALIVE_AFTER_NUL;
 	memset(&muxer, 0, sizeof(muxer));
@@ -2445,7 +2443,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)
 	if (start_async(&muxer))
 		return NULL;
 
-	ret = unpack(muxer.in, si);
+	ret = unpack(muxer.in, si, transaction);
 
 	finish_async(&muxer);
 	return ret;
@@ -2622,6 +2620,7 @@ int cmd_receive_pack(int argc,
 	struct oid_array ref = OID_ARRAY_INIT;
 	struct shallow_info si;
 	struct packet_reader reader;
+	struct odb_transaction *transaction = NULL;
 
 	struct option options[] = {
 		OPT__QUIET(&quiet, N_("quiet")),
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,
 		if (!si.nr_ours && !si.nr_theirs)
 			shallow_update = 0;
 		if (!delete_only(commands)) {
-			unpack_status = unpack_with_sideband(&si);
+			if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
+				unpack_status = "unable to start object transaction";
+			else
+				unpack_status = unpack_with_sideband(&si, transaction);
 			update_shallow_info(commands, &si, &ref);
 		}
 		use_keepalive = KEEPALIVE_ALWAYS;
-		execute_commands(commands, unpack_status, &si,
+		execute_commands(commands, unpack_status, &si, transaction,
 				 &push_options);
 		delete_tempfile(&pack_lockfile);
 		sigchain_push(SIGPIPE, SIG_IGN);
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,
 		else if (report_status)
 			report(commands, unpack_status);
 		sigchain_pop(SIGPIPE);
-		run_receive_hook(commands, "post-receive", 1,
+		run_receive_hook(commands, "post-receive", 1, NULL,
 				 &push_options);
 		run_update_post_hook(commands);
 		free_commands(commands);
Signature · 1 line
2.55.0.122.gf85a7e6620
#89 lore raw
In reply to Justin Tobler
Quoted 6 lines · Justin Tobler

Changes since V3:
- Removed ugly line break in commit message to prevent eye strain.
- odb_transaction_begin() now only sets the repository transaction on success.
- odb_transaction_env() now bubbles up error when failing to create the temporary directory.

Thanks, all the changes here look good to me and I'm happy with the state of this patch series.

Patrick