receive-pack: use ODB transactions to stage object writes
cover receive-pack: use ODB transactions to stage object writes
Greetings,
This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.
Thanks,
-Justin
Justin Tobler (6):
object-file: rename files transaction prepare function
object-file: propagate files transaction errors odb/transaction: propagate begin errors odb/transaction: propagate commit errors odb/transaction: add transaction env interface builtin/receive-pack: stage incoming objects via ODB transactions
builtin/add.c | 2 +- builtin/receive-pack.c | 46 ++++++++++-------------- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 2 +- object-file.c | 77 +++++++++++++++++++++++++++++++--------- object-file.h | 7 ++-- odb/source-files.c | 9 ++--- odb/source-inmemory.c | 3 +- odb/source-loose.c | 3 +- odb/source.h | 9 +++-- odb/transaction.c | 38 +++++++++++++++----- odb/transaction.h | 49 +++++++++++++++++++++---- read-cache.c | 2 +- 14 files changed, 173 insertions(+), 78 deletions(-)
base-commit: ab776a62a78576513ee121424adb19597fbb7613
Signature · 1 line
2.54.0.105.g59ff4886a5
PATCH 1/6 object-file: rename files transaction prepare function
The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.
Rename the function to odb_transaction_files_prepare() accordingly.
--- object-file.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
object-file.c +3 -3
@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; }; -static void prepare_loose_object_transaction (struct odb_transaction *base) +static void odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,static struct strbuf filename =STRBUF_INIT ;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -prepare_loose_object_transaction (loose->base.odb->transaction); +odb_transaction_files_prepare (loose->base.odb->transaction);odb_loose_path (loose, &filename, oid);
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,int hdrlen;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -prepare_loose_object_transaction (loose->base.odb->transaction); +odb_transaction_files_prepare (loose->base.odb->transaction);/* Since oid is not determined, save tmp file to odb path. */ strbuf_addf (&filename,"%s/" , loose->base.path);
Signature · 1 line
2.54.0.105.g59ff4886a5
Re: [PATCH 1/6] object-file: rename files transaction prepare function
Quoted 6 lines · Justin Tobler
The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via
prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.Rename the function to
odb_transaction_files_prepare()accordingly.
Taken by itself this renaming does make sense, but there are many other function that follow the historical naming convention, like {fsync,flush}_loose_object_transaction(). Should we rename them for consistency with the new naming scheme, not necessarily as part of this series but with a todo comment to do so once the dust settles, or something?
Re: [PATCH 1/6] object-file: rename files transaction prepare function
Quoted 7 lines · Junio C Hamano
Quoted 5 lines · Justin Tobler
The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via
prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.
Quoted 8 lines
Quoted 1 line
Rename the function to
odb_transaction_files_prepare()accordingly.Taken by itself this renaming does make sense, but there are many other function that follow the historical naming convention, like {fsync,flush}_loose_object_transaction(). Should we rename them for consistency with the new naming scheme, not necessarily as part of this series but with a todo comment to do so once the dust settles, or something?
Ya, both {fsync,flush}_loose_object_transaction() are probably good candidates to be renamed to odb_transaction_files_{flush,flush} also. In the next version, I'll probably add another patch to do so accordingly.
-Justin
PATCH 2/6 object-file: propagate files transaction errors
In reply to Justin ToblerThe "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle as needed.
--- object-file.c | 41 ++++++++++++++++++++++++++++------------- object-file.h | 5 +++-- odb/source-files.c | 6 +----- odb/transaction.h | 2 +- 4 files changed, 33 insertions(+), 21 deletions(-)
object-file.c +28 -13
@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; }; -static void odb_transaction_files_prepare (struct odb_transaction *base) +static int odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ; transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ; + +tmp_objdir_replace_primary_odb (transaction->objdir,0 ); + +return 0 ; } static void fsync_loose_object_transaction(struct odb_transaction *base,
@@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,/* * Cleanup after batch-mode fsync_object_files. */ -static void flush_loose_object_transaction (struct odb_transaction_files *transaction) +static int flush_loose_object_transaction (struct odb_transaction_files *transaction) {struct strbuf temp_path =STRBUF_INIT ;struct tempfile *temp;if (!transaction->objdir) -return ; +return 0 ;/* * Issue a full hardware flush against a temporary file to ensure
@@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac* Make the object files visible in the primary ODB after their data is * fully durable. */ -tmp_objdir_migrate (transaction->objdir); +if (tmp_objdir_migrate (transaction->objdir)) +return -1 ; + transaction->objdir =NULL ; + +return 0 ; }/* Finalize a file on disk, and close it. */
@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,return ret; } -static void odb_transaction_files_commit (struct odb_transaction *base) +static int odb_transaction_files_commit (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base); -flush_loose_object_transaction (transaction); +if (flush_loose_object_transaction (transaction)) +return -1 ;flush_packfile_transaction (transaction); + +return 0 ; } -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out) {struct odb_transaction_files *transaction;struct object_database *odb = source->odb; -if (odb->transaction) -return NULL ; +if (odb->transaction) { + *out =NULL ; +return 0 ; + } transaction =xcalloc (1 ,sizeof (*transaction)); transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + *out = &transaction->base; -return &transaction->base; +return 0 ; }
object-file.h +3 -2
@@ -195,8 +195,9 @@ struct odb_transaction;* Tell the object database to optimize for adding * multiple objects. odb_transaction_files_commit must be called * to make new objects visible. If a transaction is already -* pending, NULL is returned. +* pending, out is set to NULL. */ -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source); +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out);#endif /* OBJECT_FILE_H */
odb/source-files.c +1 -5
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,static int odb_source_files_begin_transaction (struct odb_source *source,struct odb_transaction **out) { -struct odb_transaction *tx =odb_transaction_files_begin (source); -if (!tx) -return -1 ; - *out = tx; -return 0 ; +return odb_transaction_files_begin (source, out); } static int odb_source_files_read_alternates(struct odb_source *source,
odb/transaction.h +1 -1
@@ -17,7 +17,7 @@ struct odb_transaction {struct odb_source *source;/* The ODB source specific callback invoked to commit a transaction. */ -void (*commit)(struct odb_transaction *transaction); +int (*commit)(struct odb_transaction *transaction);/* * This callback is expected to write the given object stream into
Signature · 1 line
2.54.0.105.g59ff4886a5
Re: [PATCH 2/6] object-file: propagate files transaction errors
Quoted 5 lines · Justin Tobler
The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the
odb_transaction_files_{prepare,begin,commit}()interfaces to allow callers to handle as needed.
Missing a then? "to handle as needed" -> "to handle them as needed"
Makes sense. It always felt a bit off that those functions didn't have a way to signal errors to the caller.
object-file.c struct odb_transaction_files { · line 505
Quoted 13 lines
object-file.c +1
-1@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; }; -static void odb_transaction_files_prepare (struct odb_transaction *base) +static int odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);
By the way, is there any reason why those functions are still hosted in "object-file.c" instead of in "odb/source-files.c"? I should probably know, but I forgot.
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518
Quoted 12 lines
object-file.c +3
-3@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ; transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ;
Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?
object-file.c static void fsync_loose_object_transaction(struct odb_transaction *base, · line 549
Quoted 6 lines
object-file.c +1
-1@@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,/* * Cleanup after batch-mode fsync_object_files. */ -static void flush_loose_object_transaction (struct odb_transaction_files *transaction) +static int flush_loose_object_transaction (struct odb_transaction_files *transaction)
Feels like this function should've been renamed in the preceding commit, as well.
object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac
Quoted 11 lines
object-file.c +1
-1{struct strbuf temp_path =STRBUF_INIT ;struct tempfile *temp;if (!transaction->objdir) -return ; +return 0 ;/* * Issue a full hardware flush against a temporary file to ensure @@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac
There is a call to xmks_tempfile() hidden that can fail, but that failure is already handled in that function itself by dying.
object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac · line 578
Quoted 6 lines
object-file.c +2
-1* Make the object files visible in the primary ODB after their data is * fully durable. */ -tmp_objdir_migrate (transaction->objdir); +if (tmp_objdir_migrate (transaction->objdir)) +return -1 ;
Feels like another case of YOLOing it. The migration could have failed, but we just ignored that failure and never told the user about it. The result may be silent corruption, I assume?
Quoted 31 lines
object-file.c +11
-5@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,return ret; } -static void odb_transaction_files_commit (struct odb_transaction *base) +static int odb_transaction_files_commit (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base); -flush_loose_object_transaction (transaction); +if (flush_loose_object_transaction (transaction)) +return -1 ;flush_packfile_transaction (transaction); + +return 0 ; } -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out) {struct odb_transaction_files *transaction;struct object_database *odb = source->odb; -if (odb->transaction) -return NULL ; +if (odb->transaction) { + *out =NULL ; +return 0 ; + }
object-file.c int read_loose_object(struct repository *repo, · line 1711
Quoted 10 lines
object-file.c +2
-1transaction =xcalloc (1 ,sizeof (*transaction)); transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + *out = &transaction->base; -return &transaction->base; +return 0 ; }
It's still somewhat fishy that we have this ODB-level transaction, but that's a preexisting issue and thus outside the scope of this patch series. Ideally though, it would be possible for there to be multiple transactions, and it would be the caller's responsibility for juggling these transactions. Just as it happens with reference transactions.
odb/transaction.h struct odb_transaction { · line 20
Quoted 10 lines
odb/transaction.h +1
-1@@ -17,7 +17,7 @@ struct odb_transaction {struct odb_source *source;/* The ODB source specific callback invoked to commit a transaction. */ -void (*commit)(struct odb_transaction *transaction); +int (*commit)(struct odb_transaction *transaction);
We might want to document the returned error code here.
Patrick
Re: [PATCH 2/6] object-file: propagate files transaction errors
Quoted 8 lines · Patrick Steinhardt
Quoted 5 lines · Justin Tobler
The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the
odb_transaction_files_{prepare,begin,commit}()interfaces to allow callers to handle as needed.Missing a then? "to handle as needed" -> "to handle them as needed"
Will fix.
[snip]
Quoted 8 lines
Quoted 8 lines
object-file.c +0
-0@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; };
Quoted 9 lines
object-file.c struct odb_transaction_files { · line 505
Quoted 5 lines
object-file.c +1
-1-static void odb_transaction_files_prepare (struct odb_transaction *base) +static int odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);By the way, is there any reason why those functions are still hosted in "
object-file.c" instead of in "odb/source-files.c"? I should probably know, but I forgot.
There are currently a couple spots in the "files" object write path in "object-file.c" that still reach into some of these transaction function that are not part of the generic ODB transaction interface. I'm hoping in a future followup series to detangle this a bit and be able to get all the "files" ODB transaction related code moved into "odb/source-files.c".
Quoted 7 lines
Quoted 7 lines
object-file.c +1
-1@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ;
Quoted 9 lines
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518
Quoted 5 lines
object-file.c +2
-2transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ;Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?
Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.
I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.
Quoted 9 lines
object-file.c static void fsync_loose_object_transaction(struct odb_transaction *base, · line 549
Quoted 6 lines
object-file.c +1
-1@@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,/* * Cleanup after batch-mode fsync_object_files. */ -static void flush_loose_object_transaction (struct odb_transaction_files *transaction) +static int flush_loose_object_transaction (struct odb_transaction_files *transaction)Feels like this function should've been renamed in the preceding commit, as well.
Ya I agree, will address in the next version.
Quoted 15 lines
object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac
Quoted 11 lines
object-file.c +1
-1{struct strbuf temp_path =STRBUF_INIT ;struct tempfile *temp;if (!transaction->objdir) -return ; +return 0 ;/* * Issue a full hardware flush against a temporary file to ensure @@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transacThere is a call to
xmks_tempfile()hidden that can fail, but that failure is already handled in that function itself by dying.
Quoted 10 lines
object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac · line 578
Quoted 6 lines
object-file.c +2
-1* Make the object files visible in the primary ODB after their data is * fully durable. */ -tmp_objdir_migrate (transaction->objdir); +if (tmp_objdir_migrate (transaction->objdir)) +return -1 ;Feels like another case of YOLOing it. The migration could have failed, but we just ignored that failure and never told the user about it. The result may be silent corruption, I assume?
Ya in this case, if the migration failed, we would just continue on which would likely result in silent corruption.
Quoted 39 lines
Quoted 39 lines
object-file.c +12
-5@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,return ret; } -static void odb_transaction_files_commit (struct odb_transaction *base) +static int odb_transaction_files_commit (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base); -flush_loose_object_transaction (transaction); +if (flush_loose_object_transaction (transaction)) +return -1 ;flush_packfile_transaction (transaction); + +return 0 ; } -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out) {struct odb_transaction_files *transaction;struct object_database *odb = source->odb; -if (odb->transaction) -return NULL ; +if (odb->transaction) { + *out =NULL ; +return 0 ; + } transaction =xcalloc (1 ,sizeof (*transaction)); transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + *out = &transaction->base;
Quoted 9 lines
object-file.c int read_loose_object(struct repository *repo, · line 1711
Quoted 3 lines
object-file.c +1
-1-return &transaction->base; +return 0 ; }It's still somewhat fishy that we have this ODB-level transaction, but that's a preexisting issue and thus outside the scope of this patch series. Ideally though, it would be possible for there to be multiple transactions, and it would be the caller's responsibility for juggling these transactions. Just as it happens with reference transactions.
I completely agree. One of the current problems preventing this is that only a single instance of tmp_objdir is allowed and stored globally. This is done to keep atexit() cleanup simple.
My plan is to eventually convert all existing tmp_objdir callsites to use ODB transactions which should hopefully allow us to remove the need for a separate tmp_objdir system. At that point, we can also work to change how temp dir cleanup is handled at exit.
Another problem is that there are also a couple of ODB transaction callsites that require to know if there is already a pending transaction for the repository and the transaction has not been wired down to these callsites. My hope is that this can be addressed though as we expand ODB transaction usage for object writes.
Quoted 7 lines
Quoted 7 lines
odb/transaction.h +0
-0@@ -17,7 +17,7 @@ struct odb_transaction {struct odb_source *source;
Quoted 5 lines
odb/transaction.h struct odb_transaction { · line 20
Quoted 3 lines
odb/transaction.h +1
-1/* The ODB source specific callback invoked to commit a transaction. */ -void (*commit)(struct odb_transaction *transaction); +int (*commit)(struct odb_transaction *transaction);We might want to document the returned error code here.
Will do in the next version.
Thanks,
-Justin
Re: [PATCH 2/6] object-file: propagate files transaction errors
Quoted 26 lines · Justin Tobler
Quoted 8 lines · Patrick Steinhardt
Quoted 7 lines · Justin Tobler
object-file.c +1
-1@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ;Quoted 9 lines
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518
Quoted 5 lines
object-file.c +2
-2transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ;Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?
Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.
Quoted 3 lines
I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.
I think I got a bit ahead of myself. The existing callers of odb_transaction_files_prepare() still continue to ignore this error. So the behavior already does remain the same here.
-Justin
Re: [PATCH 2/6] object-file: propagate files transaction errors
Quoted 27 lines · Justin Tobler
Quoted 26 lines · Justin Tobler
Quoted 8 lines · Patrick Steinhardt
Quoted 7 lines · Justin Tobler
object-file.c +1
-1@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ;Quoted 9 lines
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518
Quoted 5 lines
object-file.c +2
-2transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ;Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?
Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.
Quoted 7 lines
Quoted 3 lines
I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.
I think I got a bit ahead of myself. The existing callers of
odb_transaction_files_prepare()still continue to ignore this error. So the behavior already does remain the same here.
Oh, well, okay. I think this behaviour is plain bad -- if the caller wants to have a transaction, then we should bail in case we cannot create one. But this doesn't need to be fixed in this patch series.
Patrick
Re: [PATCH 2/6] object-file: propagate files transaction errors
Quoted 32 lines · Patrick Steinhardt
Quoted 31 lines · Justin Tobler
Quoted 29 lines · Justin Tobler
Quoted 8 lines · Patrick Steinhardt
Quoted 7 lines · Justin Tobler
object-file.c +1
-1@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ;Quoted 9 lines
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518
Quoted 5 lines
object-file.c +2
-2transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ;Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?
Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.
I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.
Quoted 7 lines
Quoted 3 lines
I think I got a bit ahead of myself. The existing callers of
odb_transaction_files_prepare()still continue to ignore this error. So the behavior already does remain the same here.Oh, well, okay. I think this behaviour is plain bad -- if the caller wants to have a transaction, then we should bail in case we cannot create one. But this doesn't need to be fixed in this patch series.
Ya, I tend to agree. The problem here is that odb_transaction_files_prepare() is being invoked lazily during the object write. This would be another argument against lazily creating the temporary directory though. In a followup series I'll explore removing this and investigate if it has any meaninful performance implications.
-Justin
Re: [PATCH 2/6] object-file: propagate files transaction errors
In reply to Justin ToblerQuoted 20 lines · Justin Tobler
Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · Justin Tobler
object-file.c +0
-0@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; };Quoted 9 lines
object-file.c struct odb_transaction_files { · line 505
Quoted 5 lines
object-file.c +1
-1-static void odb_transaction_files_prepare (struct odb_transaction *base) +static int odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);By the way, is there any reason why those functions are still hosted in "
object-file.c" instead of in "odb/source-files.c"? I should probably know, but I forgot.
Quoted 6 lines
There are currently a couple spots in the "files" object write path in "
object-file.c" that still reach into some of these transaction function that are not part of the generic ODB transaction interface. I'm hoping in a future followup series to detangle this a bit and be able to get all the "files" ODB transaction related code moved into "odb/source-files.c".
Makes sense. I also revisited that code a couple days ago, and the answer is "it's messy right now". Hopefully this will become easier to detangle as we make progress on pluggifying transactions.
Quoted 24 lines
Quoted 7 lines
Quoted 7 lines
object-file.c +1
-1@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ;Quoted 9 lines
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518
Quoted 5 lines
object-file.c +2
-2transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ;Huh. So previously we just didn't handle this error at all and just continued to tag along? Did that result in anything sensible or was this just YOLOing it?
Good question. Previously if there was an error, we wouldn't end up creating any tmpdir and would instead continue to use the primary ODB to write objects in. This change would make it a hard error if we fail to create the temp dir. This matches the behavior that git-receive-pack(1) expects, but I didn't consider that the existing callers could transparently handle there being no temp dir.
Quoted 3 lines
I suspect we may want existing ODB transaction users to continue being resilient in the same manner. In the next version, I'll maintain the same behavior.
Honestly I'd say that the change is a good one. I cannot think of a single reason to just blindly not create the transaction and proceed. But it certainly is something that should be documented as part of the commit message.
Quoted 48 lines
Quoted 38 lines
Quoted 38 lines
object-file.c +12
-5@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,return ret; } -static void odb_transaction_files_commit (struct odb_transaction *base) +static int odb_transaction_files_commit (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base); -flush_loose_object_transaction (transaction); +if (flush_loose_object_transaction (transaction)) +return -1 ;flush_packfile_transaction (transaction); + +return 0 ; } -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out) {struct odb_transaction_files *transaction;struct object_database *odb = source->odb; -if (odb->transaction) -return NULL ; +if (odb->transaction) { + *out =NULL ; +return 0 ; + } transaction =xcalloc (1 ,sizeof (*transaction)); transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + *out = &transaction->base;Quoted 9 lines
object-file.c int read_loose_object(struct repository *repo, · line 1711
Quoted 3 lines
object-file.c +1
-1-return &transaction->base; +return 0 ; }It's still somewhat fishy that we have this ODB-level transaction, but that's a preexisting issue and thus outside the scope of this patch series. Ideally though, it would be possible for there to be multiple transactions, and it would be the caller's responsibility for juggling these transactions. Just as it happens with reference transactions.
Quoted 8 lines
I completely agree. One of the current problems preventing this is that only a single instance of
tmp_objdiris allowed and stored globally. This is done to keepatexit()cleanup simple.My plan is to eventually convert all existing
tmp_objdircallsites to use ODB transactions which should hopefully allow us to remove the need for a separatetmp_objdirsystem. At that point, we can also work to change how temp dir cleanup is handled at exit.
Great.
Quoted 5 lines
Another problem is that there are also a couple of ODB transaction callsites that require to know if there is already a pending transaction for the repository and the transaction has not been wired down to these callsites. My hope is that this can be addressed though as we expand ODB transaction usage for object writes.
Yeah, agreed. Making the use of transactions explicit feels sensible to me.
Patrick
Re: [PATCH 2/6] object-file: propagate files transaction errors
In reply to Justin ToblerQuoted 5 lines · Justin Tobler
The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the
odb_transaction_files_{prepare,begin,commit}()interfaces to allow callers to handle as needed.
"handle them as needed", perhaps.
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 516
Quoted 13 lines
object-file.c +2
-2-static void odb_transaction_files_prepare (struct odb_transaction *base) +static int odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ; transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" );
If this fails, NULL is returned, and ...
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518
Quoted 4 lines
object-file.c +2
-2-if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ;
... we return -1 from here to signal an error now.
But callers of this function in write_loose_object(), and odb_source_loose_write_stream() are not prepared to react to such an error.
I guess this is nothing new. The callers ignored such an error from here in the original and proceeded writing the primary ODB anyway, and we continue to do so after this step.
Quoted 18 lines
object-file.c +2
-2@@ -542,13 +546,13 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,/* * Cleanup after batch-mode fsync_object_files. */ -static void flush_loose_object_transaction (struct odb_transaction_files *transaction) +static int flush_loose_object_transaction (struct odb_transaction_files *transaction) {struct strbuf temp_path =STRBUF_INIT ;struct tempfile *temp;if (!transaction->objdir) -return ; +return 0 ;/* * Issue a full hardware flush against a temporary file to ensure @@ -570,8 +574,12 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac* Make the object files visible in the primary ODB after their data is
object-file.c static void flush_loose_object_transaction(struct odb_transaction_files *transac · line 583
Quoted 10 lines
object-file.c +5
-1* fully durable. */ -tmp_objdir_migrate (transaction->objdir); +if (tmp_objdir_migrate (transaction->objdir)) +return -1 ; + transaction->objdir =NULL ; + +return 0 ; }
The caller of this function does react to a failure of it, ...
Quoted 7 lines
object-file.c +1
-1@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,return ret; } -static void odb_transaction_files_commit (struct odb_transaction *base) +static int odb_transaction_files_commit (struct odb_transaction *base) {
object-file.c int read_loose_object(struct repository *repo, · line 1691
Quoted 10 lines
object-file.c +4
-1struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base); -flush_loose_object_transaction (transaction); +if (flush_loose_object_transaction (transaction)) +return -1 ;flush_packfile_transaction (transaction); + +return 0 ; }
... like this, which is good. Do we need an explicit "abort-transaction", or is that implicit?
Thanks.
Re: [PATCH 2/6] object-file: propagate files transaction errors
Quoted 9 lines · Junio C Hamano
Quoted 5 lines · Justin Tobler
The "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the
odb_transaction_files_{prepare,begin,commit}()interfaces to allow callers to handle as needed."handle them as needed", perhaps.
Will fix, thanks
[snip]
Quoted 12 lines
The caller of this function does react to a failure of it, ...
Quoted 10 lines
object-file.c +1
-1@@ -1670,27 +1678,34 @@ int read_loose_object(struct repository *repo,return ret; } -static void odb_transaction_files_commit (struct odb_transaction *base) +static int odb_transaction_files_commit (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base);
Quoted 10 lines
object-file.c int read_loose_object(struct repository *repo, · line 1691
Quoted 7 lines
object-file.c +4
-1-flush_loose_object_transaction (transaction); +if (flush_loose_object_transaction (transaction)) +return -1 ;flush_packfile_transaction (transaction); + +return 0 ; }... like this, which is good. Do we need an explicit "abort-transaction", or is that implicit?
So this is currently handled implicitly via tmp-objdir.c:remove_tmp_objdir() which gets registered as an atexit() handler. As long as the tmp_objdir global remains set, it will automatically get cleaned up.
In a subsequent series, I do plan to add odb_transaction_abort() to the transaction interface. It may make sense to also use that here to make the cleanup a bit more explicit though.
-Justin
PATCH 3/6 odb/transaction: propagate begin errors
In reply to Justin ToblerWhen odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.
In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.
--- builtin/add.c | 2 +- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 2 +- object-file.c | 3 ++- odb/transaction.c | 16 +++++++++++----- odb/transaction.h | 19 +++++++++++++++---- read-cache.c | 2 +- 8 files changed, 33 insertions(+), 15 deletions(-)
builtin/add.c +1 -1
@@ -581,7 +581,7 @@ int cmd_add(int argc,string_list_clear (&only_match_skip_worktree,0 ); } - transaction =odb_transaction_begin (repo->objects); +odb_transaction_begin_or_die (repo->objects, &transaction); ps_matched =xcalloc (pathspec.nr,1 );if (add_renormalize)
builtin/unpack-objects.c +1 -1
@@ -598,7 +598,7 @@ static void unpack_all(void) progress =start_progress (the_repository,_ ("Unpacking objects" ), nr_objects);CALLOC_ARRAY (obj_list, nr_objects); - transaction =odb_transaction_begin (the_repository->objects); +odb_transaction_begin_or_die (the_repository->objects, &transaction);for (i =0 ; i < nr_objects; i++) {unpack_one (i);display_progress (progress, i +1 );
builtin/update-index.c +1 -1
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,* Allow the object layer to optimize adding multiple objects in * a batch. */ - transaction =odb_transaction_begin (the_repository->objects); +odb_transaction_begin_or_die (the_repository->objects, &transaction);while (ctx.argc) {if (parseopt_state !=PARSE_OPT_DONE ) parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
@@ -490,7 +490,7 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo); - transaction =odb_transaction_begin (the_repository->objects); +odb_transaction_begin_or_die (the_repository->objects, &transaction); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags);odb_transaction_commit (transaction);
object-file.c +2 -1
@@ -1389,8 +1389,9 @@ int index_fd(struct index_state *istate, struct object_id *oid,if (flags &INDEX_WRITE_OBJECT ) {struct object_database *odb = the_repository->objects; -struct odb_transaction *transaction =odb_transaction_begin (odb); +struct odb_transaction *transaction; +odb_transaction_begin_or_die (odb, &transaction); ret = odb_transaction_write_object_stream(odb->transaction, &stream, xsize_t(st->st_size),
odb/transaction.c +11 -5
@@ -2,14 +2,20 @@#include "odb/source.h" #include "odb/transaction.h" -struct odb_transaction *odb_transaction_begin (struct object_database *odb) +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out) { -if (odb->transaction) -return NULL ; +int ret; -odb_source_begin_transaction (odb->sources, &odb->transaction); +if (odb->transaction) { + *out =NULL ; +return 0 ; + } -return odb->transaction; + ret =odb_source_begin_transaction (odb->sources, out); + odb->transaction = *out; + +return ret; }void odb_transaction_commit (struct odb_transaction *transaction)
odb/transaction.h +15 -4
@@ -1,6 +1,8 @@#ifndef ODB_TRANSACTION_H #define ODB_TRANSACTION_H +#include "git-compat-util.h" +#include "gettext.h" #include "odb.h" #include "odb/source.h"
@@ -33,11 +35,20 @@ struct odb_transaction { };/* -* Starts an ODB transaction. Subsequent objects are written to the transaction -* and not committed until odb_transaction_commit() is invoked on the -* transaction. If the ODB already has a pending transaction, NULL is returned. +* Starts an ODB transaction and returns it via `out`. Subsequent objects are +* written to the transaction and not committed until odb_transaction_commit() +* is invoked on the transaction. Returns 0 on success and a negative value on +* error. If the ODB already has a pending transaction, `out` is set to NULL. */ -struct odb_transaction *odb_transaction_begin (struct object_database *odb); +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out); + +static inline void odb_transaction_begin_or_die (struct object_database *odb, +struct odb_transaction **out) +{ +if (odb_transaction_begin (odb, out)) +die (_ ("failed to start ODB transaction" )); +}/* * Commits an ODB transaction making the written objects visible. If the
read-cache.c +1 -1
@@ -4042,7 +4042,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,* This function is invoked from commands other than 'add', which * may not have their own transaction active. */ - transaction =odb_transaction_begin (repo->objects); +odb_transaction_begin_or_die (repo->objects, &transaction);run_diff_files (&rev,DIFF_RACY_IS_MODIFIED );odb_transaction_commit (transaction);
Signature · 1 line
2.54.0.105.g59ff4886a5
Re: [PATCH 3/6] odb/transaction: propagate begin errors
Quoted 11 lines · Justin Tobler
odb/transaction.c +2
-1@@ -2,14 +2,20 @@#include "odb/source.h" #include "odb/transaction.h" -struct odb_transaction *odb_transaction_begin (struct object_database *odb) +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out)
odb/transaction.c line 13
Quoted 10 lines
odb/transaction.c +5
-3{ -if (odb->transaction) -return NULL ; +int ret; -odb_source_begin_transaction (odb->sources, &odb->transaction); +if (odb->transaction) { + *out =NULL ; +return 0 ; + }
Hm. So we may return successful, but not set the out pointer to a transaction. And...
Quoted 16 lines
odb/transaction.h +4
-3@@ -33,11 +35,20 @@ struct odb_transaction { };/* -* Starts an ODB transaction. Subsequent objects are written to the transaction -* and not committed until odb_transaction_commit() is invoked on the -* transaction. If the ODB already has a pending transaction, NULL is returned. +* Starts an ODB transaction and returns it via `out`. Subsequent objects are +* written to the transaction and not committed until odb_transaction_commit() +* is invoked on the transaction. Returns 0 on success and a negative value on +* error. If the ODB already has a pending transaction, `out` is set to NULL. */
odb/transaction.h struct odb_transaction { · line 51
Quoted 10 lines
odb/transaction.h +9
-1-struct odb_transaction *odb_transaction_begin (struct object_database *odb); +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out); + +static inline void odb_transaction_begin_or_die (struct object_database *odb, +struct odb_transaction **out) +{ +if (odb_transaction_begin (odb, out)) +die (_ ("failed to start ODB transaction" )); +}
... we don't special-case that here, either. So a caller may invoke the function, not die, but it might still not have a valid transaction. That feels wrong to me.
Patrick
Re: [PATCH 3/6] odb/transaction: propagate begin errors
Quoted 48 lines · Patrick Steinhardt
Quoted 11 lines · Justin Tobler
odb/transaction.c +2
-1@@ -2,14 +2,20 @@#include "odb/source.h" #include "odb/transaction.h" -struct odb_transaction *odb_transaction_begin (struct object_database *odb) +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out)odb/transaction.c line 13
Quoted 10 lines
odb/transaction.c +5
-3{ -if (odb->transaction) -return NULL ; +int ret; -odb_source_begin_transaction (odb->sources, &odb->transaction); +if (odb->transaction) { + *out =NULL ; +return 0 ; + }Hm. So we may return successful, but not set the
outpointer to a transaction. And...Quoted 22 lines
odb/transaction.h +9
-4@@ -33,11 +35,20 @@ struct odb_transaction { };/* -* Starts an ODB transaction. Subsequent objects are written to the transaction -* and not committed until odb_transaction_commit() is invoked on the -* transaction. If the ODB already has a pending transaction, NULL is returned. +* Starts an ODB transaction and returns it via `out`. Subsequent objects are +* written to the transaction and not committed until odb_transaction_commit() +* is invoked on the transaction. Returns 0 on success and a negative value on +* error. If the ODB already has a pending transaction, `out` is set to NULL. */ -struct odb_transaction *odb_transaction_begin (struct object_database *odb); +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out); + +static inline void odb_transaction_begin_or_die (struct object_database *odb, +struct odb_transaction **out)
Quoted 8 lines
odb/transaction.h struct odb_transaction { · line 51
Quoted 4 lines
odb/transaction.h +4
-0+{ +if (odb_transaction_begin (odb, out)) +die (_ ("failed to start ODB transaction" )); +}... we don't special-case that here, either. So a caller may invoke the function, not die, but it might still not have a valid transaction. That feels wrong to me.
Ya, the original idea for a NULL transaction to signal that there is already a transaction in flight and nested ODB transaction users to use that to determine whether they needed to start a new transaction or not. I completely agree thought that this is rather ackward.
Instead we could just make the callers that are worried about potentially nested transactions handle this explicitly. I'll do that in the next version.
-Justin
PATCH 4/6 odb/transaction: propagate commit errors
In reply to Justin ToblerWhen odb_transaction_commit() is invoked, the return value of the backend commit callback is silently discarded. A backend has no way to signal that committing failed, such as when the "files" backend cannot migrate its temporary object directory into the permanent ODB.
In a subsequent commit, git-receive-pack(1) starts using ODB transaction to stage objects and consequently cares about such failures so it can handle the error appropriately. Change the commit callback signature to return an int error code and have odb_transaction_commit() forward it accordingly.
--- odb/transaction.c | 13 ++++++++++--- odb/transaction.h | 2 +- 2 files changed, 11 insertions(+), 4 deletions(-)
odb/transaction.c +10 -3
@@ -18,19 +18,26 @@ int odb_transaction_begin(struct object_database *odb,return ret; } -void odb_transaction_commit (struct odb_transaction *transaction) +int odb_transaction_commit (struct odb_transaction *transaction) { +int ret; +if (!transaction) -return ; +return 0 ;/* * Ensure the transaction ending matches the pending transaction. */ ASSERT (transaction == transaction->source->odb->transaction); - transaction->commit (transaction); + ret = transaction->commit (transaction); +if (ret) +return ret; + transaction->source->odb->transaction =NULL ;free (transaction); + +return 0 ; } int odb_transaction_write_object_stream(struct odb_transaction *transaction,
odb/transaction.h +1 -1
@@ -54,7 +54,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,* Commits an ODB transaction making the written objects visible. If the * specified transaction is NULL, the function is a no-op. */ -void odb_transaction_commit (struct odb_transaction *transaction); +int odb_transaction_commit (struct odb_transaction *transaction);/* * Writes the object in the provided stream into the transaction. The resulting
Signature · 1 line
2.54.0.105.g59ff4886a5
Re: [PATCH 4/6] odb/transaction: propagate commit errors
odb/transaction.h static inline void odb_transaction_begin_or_die(struct object_database *odb, · line 57
Quoted 10 lines · Justin Tobler
odb/transaction.h +1
-1@@ -54,7 +54,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,* Commits an ODB transaction making the written objects visible. If the * specified transaction is NULL, the function is a no-op. */ -void odb_transaction_commit (struct odb_transaction *transaction); +int odb_transaction_commit (struct odb_transaction *transaction);
Should the function comment be amended, as well? We should definitely point out that calling this with a NULL transaction also returns success.
Patrick
Re: [PATCH 4/6] odb/transaction: propagate commit errors
Quoted 15 lines · Patrick Steinhardt
odb/transaction.h static inline void odb_transaction_begin_or_die(struct object_database *odb, · line 57
Quoted 10 lines · Justin Tobler
odb/transaction.h +1
-1@@ -54,7 +54,7 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb,* Commits an ODB transaction making the written objects visible. If the * specified transaction is NULL, the function is a no-op. */ -void odb_transaction_commit (struct odb_transaction *transaction); +int odb_transaction_commit (struct odb_transaction *transaction);Should the function comment be amended, as well? We should definitely point out that calling this with a NULL transaction also returns success.
Will do in the next version.
Thanks,
-Justin
PATCH 5/6 odb/transaction: add transaction env interface
In reply to Justin ToblerThe ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need to access to uncommitted objects or write new objects in the staging area though.
Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction staging area.
--- object-file.c | 11 +++++++++++ odb/transaction.c | 8 ++++++++ odb/transaction.h | 19 +++++++++++++++++++ 3 files changed, 38 insertions(+)
object-file.c +11 -0
@@ -1691,6 +1691,16 @@ static int odb_transaction_files_commit(struct odb_transaction *base)return 0 ; } +static const char **odb_transaction_files_env (struct odb_transaction *base) +{ +struct odb_transaction_files *transaction = + container_of(base, struct odb_transaction_files, base); + +odb_transaction_files_prepare (&transaction->base); + +return tmp_objdir_env (transaction->objdir); +} +int odb_transaction_files_begin (struct odb_source *source,struct odb_transaction **out) {
@@ -1706,6 +1716,7 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + transaction->base.env = odb_transaction_files_env; *out = &transaction->base;return 0 ;
odb/transaction.c +8 -0
@@ -46,3 +46,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction, {return transaction->write_object_stream (transaction, stream, len, oid); } + +const char **odb_transaction_env (struct odb_transaction *transaction) +{ +if (!transaction) +return NULL ; + +return transaction->env (transaction); +}
odb/transaction.h +19 -0
@@ -32,6 +32,16 @@ struct odb_transaction {int (*write_object_stream)(struct odb_transaction *transaction,struct odb_write_stream *stream,size_t len,struct object_id *oid); + +/* +* This callback is expected to return a NULL-terminated array of +* environment variables that a child process should inherit so +* that its object writes participate in the transaction. The +* returned array is owned by the backend and remains valid until +* the transaction ends. May return NULL when the backend does not +* need to expose any state to child processes. +*/ +const char **(*env)(struct odb_transaction *transaction); };/*
@@ -65,4 +75,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,struct odb_write_stream *stream, size_t len, struct object_id *oid); +/* +* Returns a NULL-terminated array of environment variables that a child +* process should inherit so that its object writes participate in the +* transaction, suitable for passing via child_process.env. Returns NULL if +* the transaction is NULL or the backend does not expose any state to child +* processes. +*/ +const char **odb_transaction_env (struct odb_transaction *transaction); +#endif
Signature · 1 line
2.54.0.105.g59ff4886a5
Re: [PATCH 5/6] odb/transaction: add transaction env interface
Quoted 3 lines · Justin Tobler
The ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need to access to uncommitted objects or write new objects in the
s/may need to access to/may need access to/
Quoted 5 lines
staging area though.
Introduce
odb_transaction_env()which is expected to provide the set of environment variables needed by a child process to access the transaction staging area.
Possessive s is missing, I think.
Quoted 11 lines
object-file.c +3
-0@@ -1691,6 +1691,16 @@ static int odb_transaction_files_commit(struct odb_transaction *base)return 0 ; } +static const char **odb_transaction_files_env (struct odb_transaction *base) +{ + struct odb_transaction_files *transaction =
object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1706
Quoted 10 lines
object-file.c +7
-0+ container_of(base, struct odb_transaction_files, base); + +odb_transaction_files_prepare (&transaction->base); + +return tmp_objdir_env (transaction->objdir); +} +int odb_transaction_files_begin (struct odb_source *source,struct odb_transaction **out) {
Makes sense. Transactions may have a different way to quarantine the write than using a quarantine directory. So making this functionality pluggable so that backends may expose a separate set of environment variables feels sensible.
Quoted 8 lines
odb/transaction.h +0
-0@@ -32,6 +32,16 @@ struct odb_transaction {int (*write_object_stream)(struct odb_transaction *transaction,struct odb_write_stream *stream,size_t len,struct object_id *oid);
odb/transaction.h struct odb_transaction { · line 44
Quoted 10 lines
odb/transaction.h +10
-0+ +/* +* This callback is expected to return a NULL-terminated array of +* environment variables that a child process should inherit so +* that its object writes participate in the transaction. The +* returned array is owned by the backend and remains valid until +* the transaction ends. May return NULL when the backend does not +* need to expose any state to child processes. +*/ +const char **(*env)(struct odb_transaction *transaction);
Would it make more sense to adapt this function so that:
- It receives a struct strvec as input that the environment variables are to be amended to.
- It returns a normal error code to indicate errors?
Patrick
Re: [PATCH 5/6] odb/transaction: add transaction env interface
Quoted 6 lines · Patrick Steinhardt
Quoted 3 lines · Justin Tobler
The ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need to access to uncommitted objects or write new objects in the
s/may need to access to/may need access to/
Will fix.
Quoted 7 lines
Quoted 5 lines
staging area though.
Introduce
odb_transaction_env()which is expected to provide the set of environment variables needed by a child process to access the transaction staging area.Possessive s is missing, I think.
Yes. :)
[snip]
Quoted 13 lines
odb/transaction.h struct odb_transaction { · line 44
Quoted 10 lines
odb/transaction.h +10
-0+ +/* +* This callback is expected to return a NULL-terminated array of +* environment variables that a child process should inherit so +* that its object writes participate in the transaction. The +* returned array is owned by the backend and remains valid until +* the transaction ends. May return NULL when the backend does not +* need to expose any state to child processes. +*/ +const char **(*env)(struct odb_transaction *transaction);Would it make more sense to adapt this function so that:
Quoted 4 lines
- It receives a
struct strvecas input that the environment variables are to be amended to.- It returns a normal error code to indicate errors?
Ya, that is probably a more sensible interface as it would be nice to be able to signal an error. Will do in the next version.
Thanks,
-Justin
PATCH 6/6 builtin/receive-pack: stage incoming objects via ODB transactions
In reply to Justin ToblerObjects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.
Note that the temporary directory created for git-receive-pack(1) is eagerly created and uses a different prefix name. This behavior is special cased in the "files" backend by having odb_transaction_begin() callers that require this behavior provide an ODB_TRANSACTION_RECEIVE flag.
--- builtin/add.c | 2 +- builtin/receive-pack.c | 46 ++++++++++++++++------------------------ builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 2 +- object-file.c | 22 ++++++++++++++++--- object-file.h | 4 +++- odb/source-files.c | 5 +++-- odb/source-inmemory.c | 3 ++- odb/source-loose.c | 3 ++- odb/source.h | 9 +++++--- odb/transaction.c | 5 +++-- odb/transaction.h | 13 ++++++++---- read-cache.c | 2 +- 14 files changed, 70 insertions(+), 50 deletions(-)
builtin/add.c +1 -1
@@ -581,7 +581,7 @@ int cmd_add(int argc,string_list_clear (&only_match_skip_worktree,0 ); } -odb_transaction_begin_or_die (repo->objects, &transaction); +odb_transaction_begin_or_die (repo->objects, &transaction,0 ); ps_matched =xcalloc (pathspec.nr,1 );if (add_renormalize)
builtin/receive-pack.c +18 -28
129 lines not on this page · raw message
builtin/unpack-objects.c +1 -1
@@ -598,7 +598,7 @@ static void unpack_all(void) progress =start_progress (the_repository,_ ("Unpacking objects" ), nr_objects);CALLOC_ARRAY (obj_list, nr_objects); -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 );for (i =0 ; i < nr_objects; i++) {unpack_one (i);display_progress (progress, i +1 );
builtin/update-index.c +1 -1
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,* Allow the object layer to optimize adding multiple objects in * a batch. */ -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 );while (ctx.argc) {if (parseopt_state !=PARSE_OPT_DONE ) parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
@@ -490,7 +490,7 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo); -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 ); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags);odb_transaction_commit (transaction);
object-file.c +19 -3
@@ -497,6 +497,7 @@ struct odb_transaction_files {struct tmp_objdir *objdir;struct transaction_packfile packfile; +const char *prefix; };static int odb_transaction_files_prepare (struct odb_transaction *base)
@@ -513,7 +514,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)if (!transaction || transaction->objdir)return 0 ; - transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); + transaction->objdir =tmp_objdir_create (base->source->odb->repo, transaction->prefix);if (!transaction->objdir)return -1 ;
@@ -1391,7 +1392,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,struct object_database *odb = the_repository->objects;struct odb_transaction *transaction; -odb_transaction_begin_or_die (odb, &transaction); +odb_transaction_begin_or_die (odb, &transaction,0 ); ret = odb_transaction_write_object_stream(odb->transaction, &stream, xsize_t(st->st_size),
@@ -1702,7 +1703,8 @@ static const char **odb_transaction_files_env(struct odb_transaction *base) }int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {struct odb_transaction_files *transaction;struct object_database *odb = source->odb;
@@ -1717,6 +1719,20 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; transaction->base.env = odb_transaction_files_env; + + transaction->prefix ="bulk-fsync" ; +if (flags &ODB_TRANSACTION_RECEIVE ) { +/* +* ODB transactions for git-receive-pack(1) eagerly create a +* temporary directory and use a different prefix. +*/ + transaction->prefix ="incoming" ; +if (odb_transaction_files_prepare (&transaction->base)) { +free (transaction); +return -1 ; + } + } + *out = &transaction->base;return 0 ;
object-file.h +3 -1
@@ -5,6 +5,7 @@#include "object.h" #include "odb.h" #include "odb/source-loose.h" +#include "odb/transaction.h" /* The maximum size for an object header. */ #define MAX_HEADER_LEN 32
@@ -198,6 +199,7 @@ struct odb_transaction;* pending, out is set to NULL. */ int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);#endif /* OBJECT_FILE_H */
odb/source-files.c +3 -2
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source, }static int odb_source_files_begin_transaction (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -return odb_transaction_files_begin (source, out); +return odb_transaction_files_begin (source, out, flags); } static int odb_source_files_read_alternates(struct odb_source *source,
odb/source-inmemory.c +2 -1
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source, } static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED, - struct odb_transaction **out UNUSED) + struct odb_transaction **out UNUSED, + enum odb_transaction_flags flags UNUSED) {return error ("in-memory source does not support transactions" ); }
odb/source-loose.c +2 -1
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source, } static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED, - struct odb_transaction **out UNUSED) + struct odb_transaction **out UNUSED, + enum odb_transaction_flags flags UNUSED) {/* TODO: this is a known omission that we'll want to address eventually. */ return error ("loose source does not support transactions" );
odb/source.h +6 -3
@@ -3,6 +3,7 @@#include "object.h" #include "odb.h" +#include "odb/transaction.h" enum odb_source_type {/*
@@ -228,7 +229,8 @@ struct odb_source {* negative error code otherwise. */ int (*begin_transaction)(struct odb_source *source, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);/* * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,* Returns 0 on success, a negative error code otherwise. */ static inline int odb_source_begin_transaction (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -return source->begin_transaction (source, out); +return source->begin_transaction (source, out, flags); }#endif
odb/transaction.c +3 -2
@@ -3,7 +3,8 @@#include "odb/transaction.h" int odb_transaction_begin (struct object_database *odb, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {int ret;
@@ -12,7 +13,7 @@ int odb_transaction_begin(struct object_database *odb,return 0 ; } - ret =odb_source_begin_transaction (odb->sources, out); + ret =odb_source_begin_transaction (odb->sources, out, flags); odb->transaction = *out;return ret;
odb/transaction.h +9 -4
@@ -4,7 +4,6 @@#include "git-compat-util.h" #include "gettext.h" #include "odb.h" -#include "odb/source.h" /* * A transaction may be started for an object database prior to writing new
@@ -44,6 +43,10 @@ struct odb_transaction {const char **(*env)(struct odb_transaction *transaction); }; +enum odb_transaction_flags { +ODB_TRANSACTION_RECEIVE = (1 <<0 ), +}; +/* * Starts an ODB transaction and returns it via `out`. Subsequent objects are * written to the transaction and not committed until odb_transaction_commit()
@@ -51,12 +54,14 @@ struct odb_transaction {* error. If the ODB already has a pending transaction, `out` is set to NULL. */ int odb_transaction_begin (struct object_database *odb, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);static inline void odb_transaction_begin_or_die (struct object_database *odb, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -if (odb_transaction_begin (odb, out)) +if (odb_transaction_begin (odb, out, flags))die (_ ("failed to start ODB transaction" )); }
read-cache.c +1 -1
@@ -4042,7 +4042,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,* This function is invoked from commands other than 'add', which * may not have their own transaction active. */ -odb_transaction_begin_or_die (repo->objects, &transaction); +odb_transaction_begin_or_die (repo->objects, &transaction,0 );run_diff_files (&rev,DIFF_RACY_IS_MODIFIED );odb_transaction_commit (transaction);
Signature · 1 line
2.54.0.105.g59ff4886a5
Re: [PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions
Quoted 10 lines · Justin Tobler
Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through
tmp_objdirdirectly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.Note that the temporary directory created for git-receive-pack(1) is eagerly created and uses a different prefix name. This behavior is
A different prefix name compared to what?
Quoted 3 lines
special cased in the "files" backend by having
odb_transaction_begin()callers that require this behavior provide anODB_TRANSACTION_RECEIVEflag.
Okay. I guess this is to retain existing behaviour where the temporary directory is created lazily everywhere else. Makes me wonder whether we should eventually change this to just unconditionally create the directory in all cases so that we can drop this new flag.
It might've also made sense to split this commit up into two: one to introduce the flag parameter, and then one to do the changes to git-receive-pack(1).
builtin/receive-pack.c static enum { · line 117
Quoted 13 lines
builtin/receive-pack.c +0
-2@@ -112,8 +112,6 @@ static enum { } use_keepalive;static int keepalive_in_sec =5 ; -static struct tmp_objdir *tmp_objdir; - static struct proc_receive_ref { unsigned int want_add:1, want_delete:1,
I assume the goal is that we convert all other users of the tmp-objdir subsystem to also use transactions eventually, so that this becomes an implementation detail fo the files transaction?
builtin/receive-pack.c static void execute_commands(struct command *commands, · line 2116
Quoted 13 lines
builtin/receive-pack.c +1
-2@@ -2106,14 +2104,13 @@ static void execute_commands(struct command *commands,* Now we'll start writing out refs, which means the objects need * to be in their final positions so that other processes can see them. */ -if (tmp_objdir_migrate (tmp_objdir) <0 ) { +if (odb_transaction_commit (the_repository->objects->transaction)) {for (cmd = commands; cmd; cmd = cmd->next) {if (!cmd->error_string) cmd->error_string ="unable to migrate objects to permanent storage" ; }return ; } - tmp_objdir =NULL ;
We don't need to unset the transaction because that's what odb_transaction_commit() already does for us, I assume?
builtin/receive-pack.c static void push_header_arg(struct strvec *args, struct pack_header *hdr) · line 2330
Quoted 10 lines
builtin/receive-pack.c +2
-1@@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr) ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries)); } -static const char *unpack (int err_fd,struct shallow_info *si) +static const char *unpack (int err_fd,struct shallow_info *si, +struct odb_transaction *transaction) {struct pack_header hdr;const char *hdr_err;
It feels a bit weird that we sometimes pass the transaction as parameter, whereas othertimes we access it via the_repository.
Quoted 9 lines
builtin/receive-pack.c +0
-5@@ -2351,20 +2349,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)strvec_push (&child.args, alt_shallow_file); } - tmp_objdir =tmp_objdir_create (the_repository,"incoming" ); -if (!tmp_objdir) { -if (err_fd >0 ) -close (err_fd); -return "unable to create temporary object directory" ;
builtin/receive-pack.c static const char *unpack(int err_fd, struct shallow_info *si) · line 2352
Quoted 10 lines
builtin/receive-pack.c +1
-9- } -strvec_pushv (&child.env,tmp_objdir_env (tmp_objdir)); - -/* -* Normally we just pass the tmp_objdir environment to the child -* processes that do the heavy lifting, but we may need to see these -* objects ourselves to set up shallow information. -*/ -tmp_objdir_add_as_alternate (tmp_objdir); +strvec_pushv (&child.env,odb_transaction_env (transaction));
Interesting, this here seems like a change in behaviour. Previously we added the transactions as an alternate, but now we only propagate it via the environment. I didn't see this mentioned in the commit message.
builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2698
Quoted 7 lines
builtin/receive-pack.c +2
-1@@ -2707,7 +2694,10 @@ int cmd_receive_pack(int argc,if (!si.nr_ours && !si.nr_theirs) shallow_update =0 ;if (!delete_only (commands)) { - unpack_status =unpack_with_sideband (&si); +if (odb_transaction_begin (the_repository->objects, &transaction,ODB_TRANSACTION_RECEIVE )) + unpack_status ="unable to start ODB transaction" ;
s/ODB/object/
This may be visible to the user, and "ODB" may mean nothing to them.
Quoted 22 lines
object-file.c +6
-1@@ -1702,7 +1703,8 @@ static const char **odb_transaction_files_env(struct odb_transaction *base) }int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {struct odb_transaction_files *transaction;struct object_database *odb = source->odb;@@ -1717,6 +1719,20 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; transaction->base.env = odb_transaction_files_env; + + transaction->prefix ="bulk-fsync" ; +if (flags &ODB_TRANSACTION_RECEIVE ) { +/*
object-file.c int odb_transaction_files_begin(struct odb_source *source, · line 1735
Quoted 10 lines
object-file.c +10
-0+* ODB transactions for git-receive-pack(1) eagerly create a +* temporary directory and use a different prefix. +*/ + transaction->prefix ="incoming" ; +if (odb_transaction_files_prepare (&transaction->base)) { +free (transaction); +return -1 ; + } + } +
Okay, makes sense. I really wonder whether we need to insist this much on the exact name used by this, but better be safe than sorry for now I guess.
And as mentioned before, I also wonder whether it really makes sense to have the lazy creation of the tmp-objdir. Maybe add a NEEDSWORK item here that mentions we want to investigate whether this is even needed at all?
odb/transaction.h struct odb_transaction { · line 48
Quoted 11 lines
odb/transaction.h +3
-0@@ -44,6 +43,10 @@ struct odb_transaction {const char **(*env)(struct odb_transaction *transaction); }; +enum odb_transaction_flags { +ODB_TRANSACTION_RECEIVE = (1 <<0 ), +};
It's not clear at all what this flag does based on its name, so we should have documentation for it.
Patrick
Re: [PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions
Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · Justin Tobler
Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through
tmp_objdirdirectly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.
Quoted 4 lines
Quoted 2 lines
Note that the temporary directory created for git-receive-pack(1) is eagerly created and uses a different prefix name. This behavior is
A different prefix name compared to what?
Currently the temporary directories created for ODB transactions all use the prefix "bulk-fsync". The temp dir created by git-receive-pack(1) is expected to have the prefix "incoming".
Quoted 8 lines
Quoted 3 lines
special cased in the "files" backend by having
odb_transaction_begin()callers that require this behavior provide anODB_TRANSACTION_RECEIVEflag.Okay. I guess this is to retain existing behaviour where the temporary directory is created lazily everywhere else. Makes me wonder whether we should eventually change this to just unconditionally create the directory in all cases so that we can drop this new flag.
It would be nice to not have to have a flag here, but if we want to also keep the existing temp dir prefixes, we would also need to keep the flags.
Quoted 3 lines
It might've also made sense to split this commit up into two: one to introduce the flag parameter, and then one to do the changes to git-receive-pack(1).
Ya, I think you are right. I actually originally had it this way but squashed these two commits together for some reason. I'll split them back up in the next version.
Quoted 8 lines
Quoted 8 lines
builtin/receive-pack.c +0
-0@@ -112,8 +112,6 @@ static enum { } use_keepalive;static int keepalive_in_sec =5 ;
Quoted 9 lines
builtin/receive-pack.c static enum { · line 117
Quoted 5 lines
builtin/receive-pack.c +0
-2-static struct tmp_objdir *tmp_objdir; - static struct proc_receive_ref { unsigned int want_add:1, want_delete:1,I assume the goal is that we convert all other users of the tmp-objdir subsystem to also use transactions eventually, so that this becomes an implementation detail fo the files transaction?
Yes, that is exactly my plan :)
Quoted 8 lines
Quoted 8 lines
builtin/receive-pack.c +1
-1@@ -2106,14 +2104,13 @@ static void execute_commands(struct command *commands,* Now we'll start writing out refs, which means the objects need * to be in their final positions so that other processes can see them. */ -if (tmp_objdir_migrate (tmp_objdir) <0 ) { +if (odb_transaction_commit (the_repository->objects->transaction)) {for (cmd = commands; cmd; cmd = cmd->next) {if (!cmd->error_string)
Quoted 8 lines
builtin/receive-pack.c static void execute_commands(struct command *commands, · line 2116
Quoted 5 lines
builtin/receive-pack.c +0
-1cmd->error_string ="unable to migrate objects to permanent storage" ; }return ; } - tmp_objdir =NULL ;We don't need to unset the transaction because that's what
odb_transaction_commit()already does for us, I assume?
That is correct. The tmp_objdir global is actually removed and now competely managed by the ODB transaction. We can probably actuall remove the "tmp-objdir" include now too. I'll do that in the next version.
Quoted 4 lines
Quoted 4 lines
builtin/receive-pack.c +0
-0@@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr) ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries)); }
Quoted 9 lines
builtin/receive-pack.c static void push_header_arg(struct strvec *args, struct pack_header *hdr) · line 2330
Quoted 6 lines
builtin/receive-pack.c +2
-1-static const char *unpack (int err_fd,struct shallow_info *si) +static const char *unpack (int err_fd,struct shallow_info *si, +struct odb_transaction *transaction) {struct pack_header hdr;const char *hdr_err;It feels a bit weird that we sometimes pass the transaction as parameter, whereas othertimes we access it via
the_repository.
That's fair. I was trying to avoid the churn of wiring to all its callsites, but it's probably best to be consistent. Maybe it would be fine to just create a transaction global like we do for the reference transaction?
Quoted 15 lines
Quoted 15 lines
builtin/receive-pack.c +0
-11@@ -2351,20 +2349,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)strvec_push (&child.args, alt_shallow_file); } - tmp_objdir =tmp_objdir_create (the_repository,"incoming" ); -if (!tmp_objdir) { -if (err_fd >0 ) -close (err_fd); -return "unable to create temporary object directory" ; - } -strvec_pushv (&child.env,tmp_objdir_env (tmp_objdir)); - -/* -* Normally we just pass the tmp_objdir environment to the child -* processes that do the heavy lifting, but we may need to see these
Quoted 8 lines
builtin/receive-pack.c static const char *unpack(int err_fd, struct shallow_info *si) · line 2352
Quoted 4 lines
builtin/receive-pack.c +1
-3-* objects ourselves to set up shallow information. -*/ -tmp_objdir_add_as_alternate (tmp_objdir); +strvec_pushv (&child.env,odb_transaction_env (transaction));Interesting, this here seems like a change in behaviour. Previously we added the transactions as an alternate, but now we only propagate it via the environment. I didn't see this mentioned in the commit message.
That's not quite correct. By using the ODB transaction interface, the underlying tmp_objdir is managed transparently. When we begin the ODB transaction, the temp directory that gets created is automatically created and applied as the primary ODB. This ultimately produces an equivalent result, its just now set up when the transaction is started (which happens a little bit earlier).
The only behavior change here is that the temp directory is being applied as the primary instead of an alternate in the main git-receive-pack(1) process. Since all writes though are handled by the child processes that get spawned, this shouldn't really matter though.
This is certainly rather confusing though, and should be mentioned in the commit message. I'll do this in the next version.
Quoted 11 lines
builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2698
Quoted 7 lines
builtin/receive-pack.c +2
-1@@ -2707,7 +2694,10 @@ int cmd_receive_pack(int argc,if (!si.nr_ours && !si.nr_theirs) shallow_update =0 ;if (!delete_only (commands)) { - unpack_status =unpack_with_sideband (&si); +if (odb_transaction_begin (the_repository->objects, &transaction,ODB_TRANSACTION_RECEIVE )) + unpack_status ="unable to start ODB transaction" ;s/ODB/object/
This may be visible to the user, and "ODB" may mean nothing to them.
Will change in the next version.
Quoted 37 lines
Quoted 22 lines
object-file.c +6
-1@@ -1702,7 +1703,8 @@ static const char **odb_transaction_files_env(struct odb_transaction *base) }int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {struct odb_transaction_files *transaction;struct object_database *odb = source->odb;@@ -1717,6 +1719,20 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; transaction->base.env = odb_transaction_files_env; + + transaction->prefix ="bulk-fsync" ; +if (flags &ODB_TRANSACTION_RECEIVE ) { +/* object-file.c int odb_transaction_files_begin(struct odb_source *source, · line 1735
Quoted 10 lines
object-file.c +10
-0+* ODB transactions for git-receive-pack(1) eagerly create a +* temporary directory and use a different prefix. +*/ + transaction->prefix ="incoming" ; +if (odb_transaction_files_prepare (&transaction->base)) { +free (transaction); +return -1 ; + } + } +Okay, makes sense. I really wonder whether we need to insist this much on the exact name used by this, but better be safe than sorry for now I guess.
Quoted 4 lines
And as mentioned before, I also wonder whether it really makes sense to have the lazy creation of the tmp-objdir. Maybe add a NEEDSWORK item here that mentions we want to investigate whether this is even needed at all?
Ya, I was worried that there would be some reason the temp dir prefix should remain the same or there would be some performance reason to lazily create directories. I didn't investigate too deeply though. I'll add a NEEDSWORK comment to make note of this.
Quoted 8 lines
Quoted 8 lines
odb/transaction.h +0
-0@@ -44,6 +43,10 @@ struct odb_transaction {const char **(*env)(struct odb_transaction *transaction); };
Quoted 6 lines
odb/transaction.h struct odb_transaction { · line 48
Quoted 3 lines
odb/transaction.h +3
-0+enum odb_transaction_flags { +ODB_TRANSACTION_RECEIVE = (1 <<0 ), +};It's not clear at all what this flag does based on its name, so we should have documentation for it.
Will update.
Thanks,
-Justin
Re: [PATCH 6/6] builtin/receive-pack: stage incoming objects via ODB transactions
Quoted 23 lines · Justin Tobler
Quoted 9 lines · Patrick Steinhardt
Quoted 8 lines · Justin Tobler
Objects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through
tmp_objdirdirectly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.Quoted 4 lines
Quoted 2 lines
Note that the temporary directory created for git-receive-pack(1) is eagerly created and uses a different prefix name. This behavior is
A different prefix name compared to what?
Currently the temporary directories created for ODB transactions all use the prefix "bulk-fsync". The temp dir created by git-receive-pack(1) is expected to have the prefix "incoming".
Quoted 4 lines
Quoted 3 lines
special cased in the "files" backend by having
odb_transaction_begin()callers that require this behavior provide anODB_TRANSACTION_RECEIVEflag.
Quoted 8 lines
Quoted 4 lines
Okay. I guess this is to retain existing behaviour where the temporary directory is created lazily everywhere else. Makes me wonder whether we should eventually change this to just unconditionally create the directory in all cases so that we can drop this new flag.
It would be nice to not have to have a flag here, but if we want to also keep the existing temp dir prefixes, we would also need to keep the flags.
Fair. Makes me wonder whether we really need to keep the exact same naming for this temporary directory. This is so deep into internals that I'm not sure whether we really need to treat this as part of our interface. I'm rather inclined to say it's not necessary.
In any case, I think it's fine to defer that discussion and keep this as-is for now. But we might keep it in the back of our minds and maybe simplify this in a subsequent patch series.
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
builtin/receive-pack.c +0
-0@@ -2326,7 +2323,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr) ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries)); }Quoted 9 lines
builtin/receive-pack.c static void push_header_arg(struct strvec *args, struct pack_header *hdr) · line 2330
Quoted 6 lines
builtin/receive-pack.c +2
-1-static const char *unpack (int err_fd,struct shallow_info *si) +static const char *unpack (int err_fd,struct shallow_info *si, +struct odb_transaction *transaction) {struct pack_header hdr;const char *hdr_err;It feels a bit weird that we sometimes pass the transaction as parameter, whereas othertimes we access it via
the_repository.
Quoted 4 lines
That's fair. I was trying to avoid the churn of wiring to all its callsites, but it's probably best to be consistent. Maybe it would be fine to just create a transaction global like we do for the reference transaction?
My first kneejerk reaction was "no", but then I noticed that the global variable you're talking about is local to "builtin/receive-pack.c". So that might be an okayish solution.
Thanks!
Patrick
Quoted 10 lines · Justin Tobler
Greetings,
This patch series replaces direct usage of the
tmp_objdirinterfaces in git-receive-pack(1) to instead use theodb_transactioninterfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.
Thanks, this was a pleasant read. I've got a bunch of comments, but overall I really like the direction of this patch series.
Patrick
Quoted 11 lines · Justin Tobler
Greetings,
This patch series replaces direct usage of the
tmp_objdirinterfaces in git-receive-pack(1) to instead use theodb_transactioninterfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.
Quoted 2 lines
Thanks,
-Justin
The integration cycle this morning was somehow more painful than other cycles.
FYI, this topic had some interactions with ps/odb-source-packed and ps/refs-avoid-chdir-notify-reparent and needed the following evil-merge fix to make it build.
commit 721c15c1d5ef8f8aab8b9f50463e979e890b1ab6
Author: Junio C Hamano <[email protected]> Date: Wed Jun 24 12:45:35 2026 -0700
merge-fix/jt/receive-pack-use-odb-transactions
conflict with ps/odb-source-packed and ps/refs-avoid-chdir-notify-reparent
odb/source-packed.c +2 -1
@@ -545,7 +545,8 @@ static int odb_source_packed_write_object_stream(struct odb_source *source UNUSE } static int odb_source_packed_begin_transaction(struct odb_source *source UNUSED, - struct odb_transaction **out UNUSED) + struct odb_transaction **out UNUSED, + enum odb_transaction_flags flags UNUSED) {return error ("packed backend cannot begin transactions" ); }
t/unit-tests/u-reftable-table.c +4 -2
@@ -235,7 +235,8 @@ void test_reftable_table__seek_invalid_log_offset(void)uint8_t *footer;cl_reftable_write_to_buf (&buf, refs,ARRAY_SIZE (refs), - logs,ARRAY_SIZE (logs),NULL ); + logs,ARRAY_SIZE (logs), +REFTABLE_HASH_SHA1 ,NULL );/* * Corrupt the log section offset stored in the footer so that it points
@@ -278,7 +279,8 @@ void test_reftable_table__new_with_truncated_table(void)struct reftable_table *table;struct reftable_buf buf =REFTABLE_BUF_INIT ; -cl_reftable_write_to_buf (&buf, refs,ARRAY_SIZE (refs),NULL ,0 ,NULL ); +cl_reftable_write_to_buf (&buf, refs,ARRAY_SIZE (refs),NULL ,0 , +REFTABLE_HASH_SHA1 ,NULL );/* * Truncate the table so that it is large enough to read the header, but
v2 cover receive-pack: use ODB transactions to stage object writes
In reply to Justin ToblerGreetings,
This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.
Changes since V1:
- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling from odb_transaction_files_begin().
- odb_transaction_begin() now returns an error if there is already an inflight transaction pending instead of setting the out pointer to NULL.
- Updated odb_transaction_env() to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it from the_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.
Thanks,
-Justin
Justin Tobler (11):
object-file: rename files transaction prepare function
object-file: rename files transaction fsync function
object-file: embed transaction flush logic in commit function
object-file: drop check for inflight transactions
object-file: propagate files transaction errors odb/transaction: propagate begin errors odb/transaction: propagate commit errors odb/transaction: add transaction env interface odb/transaction: introduce ODB transaction flags builtin/receive-pack: drop redundant tmpdir env builtin/receive-pack: stage incoming objects via ODB transactions
builtin/add.c | 2 +- builtin/receive-pack.c | 69 ++++++++--------- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 7 +- object-file.c | 159 +++++++++++++++++++++++++-------------- object-file.h | 8 +- odb/source-files.c | 9 +-- odb/source-inmemory.c | 3 +- odb/source-loose.c | 3 +- odb/source.h | 9 ++- odb/transaction.c | 34 +++++++-- odb/transaction.h | 60 ++++++++++++--- read-cache.c | 7 +- 14 files changed, 244 insertions(+), 130 deletions(-)
Range-diff against v1:
1: 9c14b219ad = 1: 9c14b219ad object-file: rename files transaction prepare function -: ---------- > 2: 5703a9e93b object-file: rename files transaction fsync function -: ---------- > 3: 4c37398ac8 object-file: embed transaction flush logic in commit function -: ---------- > 4: 623c6b02ea object-file: drop check for inflight transactions 2: 201f543692 ! 5: ca59176657 object-file: propagate files transaction errors
@@ Commit message
the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow
- callers to handle as needed.
+ callers to handle them as needed.
@@ object-file.c: static void odb_transaction_files_prepare(struct odb_transaction
+ return 0;
}
- static void fsync_loose_object_transaction(struct odb_transaction *base,
-@@ object-file.c: static void fsync_loose_object_transaction(struct odb_transaction *base,
- /*
- * Cleanup after batch-mode fsync_object_files.
- */
--static void flush_loose_object_transaction(struct odb_transaction_files *transaction) -+static int flush_loose_object_transaction(struct odb_transaction_files *transaction)
- {
- struct strbuf temp_path = STRBUF_INIT;
- struct tempfile *temp;
-
- if (!transaction->objdir)
-- return;
-+ return 0;
-
- /*
- * Issue a full hardware flush against a temporary file to ensure
-@@ object-file.c: static void flush_loose_object_transaction(struct odb_transaction_files *transac
- * Make the object files visible in the primary ODB after their data is
- * fully durable.
- */
-- tmp_objdir_migrate(transaction->objdir);
-+ if (tmp_objdir_migrate(transaction->objdir))
-+ return -1;
-+
- transaction->objdir = NULL;
-+
-+ return 0;
- }
-
- /* Finalize a file on disk, and close it. */
+ static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ object-file.c: int read_loose_object(struct repository *repo,
return ret;
}
@@ object-file.c: int read_loose_object(struct repository *repo,
{
struct odb_transaction_files *transaction =
container_of(base, struct odb_transaction_files, base);
+@@ object-file.c: static void odb_transaction_files_commit(struct odb_transaction *base)
+ * Make the object files visible in the primary ODB after their data is
+ * fully durable.
+ */
+- tmp_objdir_migrate(transaction->objdir);
++ if (tmp_objdir_migrate(transaction->objdir))
++ return -1;
++
+ transaction->objdir = NULL;
+ }
-- flush_loose_object_transaction(transaction);
-+ if (flush_loose_object_transaction(transaction))
-+ return -1;
flush_packfile_transaction(transaction);
+
+ return 0;
@@ object-file.c: int read_loose_object(struct repository *repo,
+ struct odb_transaction **out)
{
struct odb_transaction_files *transaction;
- struct object_database *odb = source->odb;
-
-- if (odb->transaction)
-- return NULL;
-+ if (odb->transaction) {
-+ *out = NULL;
-+ return 0;
-+ }
- transaction = xcalloc(1, sizeof(*transaction));
+@@ object-file.c: struct odb_transaction *odb_transaction_files_begin(struct odb_source *source)
transaction->base.source = source;
transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
@@ object-file.c: int read_loose_object(struct repository *repo,
## object-file.h ##
@@ object-file.h: struct odb_transaction;
- * Tell the object database to optimize for adding
* multiple objects. odb_transaction_files_commit must be called
- * to make new objects visible. If a transaction is already
-- * pending, NULL is returned.
-+ * pending, out is set to NULL.
+ * to make new objects visible.
*/
-struct odb_transaction *odb_transaction_files_begin(struct odb_source *source); +int odb_transaction_files_begin(struct odb_source *source,
@@ odb/source-files.c: static int odb_source_files_write_object_stream(struct odb_s
## odb/transaction.h ##
@@ odb/transaction.h: struct odb_transaction {
+ /* The ODB source the transaction is opened against. */
struct odb_source *source;
- /* The ODB source specific callback invoked to commit a transaction. */
+- /* The ODB source specific callback invoked to commit a transaction. */
- void (*commit)(struct odb_transaction *transaction);
++ /*
++ * The ODB source specific callback invoked to commit a transaction.
++ * Returns 0 on success, a negative error code otherwise.
++ */
+ int (*commit)(struct odb_transaction *transaction);
/*
3: 68cdd88ab5 ! 6: 717a1ce9a7 odb/transaction: propagate begin errors
@@ Commit message
convenience for callsites that do not need to handle errors
explicitly.
+ Note that `odb_transaction_begin()` now returns an error when the ODB
+ already has an inflight transaction pending. ODB transaction call sites
+ that may encounter an inflight transaction are updated to explicitly
+ handle this case.
+
Signed-off-by: Justin Tobler <[email protected]>
## builtin/add.c ##
@@ builtin/update-index.c: int cmd_update_index(int argc,
parseopt_state = parse_options_step(&ctx, options,
## cache-tree.c ##
+@@ cache-tree.c: static int update_one(struct cache_tree *it,
+
+ int cache_tree_update(struct index_state *istate, int flags)
+ {
++ int inflight = !!the_repository->objects->transaction;
+ struct odb_transaction *transaction;
+ int skip, i;
+
@@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)
trace_performance_enter();
trace2_region_enter("cache_tree", "update", istate->repo);
- transaction = odb_transaction_begin(the_repository->objects);
-+ odb_transaction_begin_or_die(the_repository->objects, &transaction);
++ if (!inflight)
++ odb_transaction_begin_or_die(the_repository->objects, &transaction);
i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
"", 0, &skip, flags);
- odb_transaction_commit(transaction);
+- odb_transaction_commit(transaction);
++ if (!inflight)
++ odb_transaction_commit(transaction);
+ trace2_region_leave("cache_tree", "update", istate->repo);
+ trace_performance_leave("cache_tree_update");
+ if (i < 0)
## object-file.c ##
@@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
@@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
if (flags & INDEX_WRITE_OBJECT) {
struct object_database *odb = the_repository->objects;
- struct odb_transaction *transaction = odb_transaction_begin(odb);
-+ struct odb_transaction *transaction;
++ struct odb_transaction *transaction = odb->transaction;
++ int inflight = !!transaction;
-+ odb_transaction_begin_or_die(odb, &transaction);
- ret = odb_transaction_write_object_stream(odb->transaction,
+- ret = odb_transaction_write_object_stream(odb->transaction,
++ if (!inflight)
++ odb_transaction_begin_or_die(odb, &transaction);
++ ret = odb_transaction_write_object_stream(transaction,
&stream,
xsize_t(st->st_size),
+ oid);
+- odb_transaction_commit(transaction);
++ if (!inflight)
++ odb_transaction_commit(transaction);
+ } else {
+ ret = hash_blob_stream(&stream,
+ the_repository->hash_algo, oid,
## odb/transaction.c ##
@@
@@ odb/transaction.c
+int odb_transaction_begin(struct object_database *odb,
+ struct odb_transaction **out)
{
-- if (odb->transaction)
-- return NULL;
+ int ret;
++
+ if (odb->transaction)
+- return NULL;
++ return -1;
- odb_source_begin_transaction(odb->sources, &odb->transaction);
-+ if (odb->transaction) {
-+ *out = NULL;
-+ return 0;
-+ }
-
-- return odb->transaction;
+ ret = odb_source_begin_transaction(odb->sources, out);
+ odb->transaction = *out;
-+
+
+- return odb->transaction;
+ return ret;
}
@@ odb/transaction.h: struct odb_transaction {
+ * Starts an ODB transaction and returns it via `out`. Subsequent objects are
+ * written to the transaction and not committed until odb_transaction_commit()
+ * is invoked on the transaction. Returns 0 on success and a negative value on
-+ * error. If the ODB already has a pending transaction, `out` is set to NULL.
++ * error. Note that it is considered an error to start a new transaction if the
++ * ODB already has an inflight transaction pending.
*/
-struct odb_transaction *odb_transaction_begin(struct object_database *odb);
+int odb_transaction_begin(struct object_database *odb,
@@ odb/transaction.h: struct odb_transaction {
* Commits an ODB transaction making the written objects visible. If the
## read-cache.c ##
+@@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,
+ const struct pathspec *pathspec, char *ps_matched,
+ int include_sparse, int flags, int ignored_too )
+ {
++ int inflight = !!repo->objects->transaction;
+ struct odb_transaction *transaction;
+ struct update_callback_data data;
+ struct rev_info rev;
@@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,
* This function is invoked from commands other than 'add', which
* may not have their own transaction active.
*/
- transaction = odb_transaction_begin(repo->objects);
-+ odb_transaction_begin_or_die(repo->objects, &transaction);
++ if (!inflight)
++ odb_transaction_begin_or_die(repo->objects, &transaction);
run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
- odb_transaction_commit(transaction);
+- odb_transaction_commit(transaction);
++ if (!inflight)
++ odb_transaction_commit(transaction);
+ release_revisions(&rev);
+ return !!data.add_errors;
-: ---------- > 7: ff8e133965 odb/transaction: propagate commit errors
5: 82302db9f0 ! 8: 264ba94b83 odb/transaction: add transaction env interface
@@ Commit message
The ODB transaction backend is responsible for creating/managing its own
staging area for writing objects. Other child processes spawned by Git
- may need to access to uncommitted objects or write new objects in the
+ may need access to uncommitted objects or write new objects in the
staging area though.
Introduce `odb_transaction_env()` which is expected to provide the set
of environment variables needed by a child process to access the
- transaction staging area.
+ transaction's staging area.
Signed-off-by: Justin Tobler <[email protected]>
## object-file.c ##
+@@
+ #include "path.h"
+ #include "read-cache-ll.h"
+ #include "setup.h"
++#include "strvec.h"
+ #include "tempfile.h"
+ #include "tmp-objdir.h"
+
@@ object-file.c: static int odb_transaction_files_commit(struct odb_transaction *base)
return 0;
}
-+static const char **odb_transaction_files_env(struct odb_transaction *base)
++static int odb_transaction_files_env(struct odb_transaction *base,
++ struct strvec *env)
+{
+ struct odb_transaction_files *transaction =
+ container_of(base, struct odb_transaction_files, base);
+
+ odb_transaction_files_prepare(&transaction->base);
++ strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+
-+ return tmp_objdir_env(transaction->objdir);
++ return 0;
+}
+
int odb_transaction_files_begin(struct odb_source *source,
@@ odb/transaction.c: int odb_transaction_write_object_stream(struct odb_transactio
return transaction->write_object_stream(transaction, stream, len, oid);
}
+
-+const char **odb_transaction_env(struct odb_transaction *transaction)
++int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env)
+{
+ if (!transaction)
-+ return NULL;
++ return 0;
+
-+ return transaction->env(transaction);
++ return transaction->env(transaction, env);
+}
## odb/transaction.h ##
@@ odb/transaction.h: struct odb_transaction {
struct object_id *oid);
+
+ /*
-+ * This callback is expected to return a NULL-terminated array of
-+ * environment variables that a child process should inherit so
-+ * that its object writes participate in the transaction. The
-+ * returned array is owned by the backend and remains valid until
-+ * the transaction ends. May return NULL when the backend does not
-+ * need to expose any state to child processes.
++ * This callback is expected to populate the provided strvec with the
++ * environment variables that a child process should inherit so that its
++ * object writes participate in the transaction. Returns 0 on success, a
++ * negative error code otherwise.
+ */
-+ const char **(*env)(struct odb_transaction *transaction);
++ int (*env)(struct odb_transaction *transaction, struct strvec *env);
};
/*
@@ odb/transaction.h: int odb_transaction_write_object_stream(struct odb_transactio
size_t len, struct object_id *oid);
+/*
-+ * Returns a NULL-terminated array of environment variables that a child
++ * Populates the provided strvec with the environment variables that a child
+ * process should inherit so that its object writes participate in the
-+ * transaction, suitable for passing via child_process.env. Returns NULL if
-+ * the transaction is NULL or the backend does not expose any state to child
-+ * processes.
++ * transaction, suitable for using via child_process.env. Returns 0 on success,
++ * a negative error code otherwise. Note that, if the specified transaction is
++ * NULL, the function is a no-op and no error is returned.
+ */
-+const char **odb_transaction_env(struct odb_transaction *transaction);
++int odb_transaction_env(struct odb_transaction *transaction, struct strvec *env);
+
#endif 6: 0674e8a3d3 ! 9: 1e0a491ef2 builtin/receive-pack: stage incoming objects via ODB transactions
@@ Metadata
Author: Justin Tobler <[email protected]>
## Commit message ##
- builtin/receive-pack: stage incoming objects via ODB transactions
+ odb/transaction: introduce ODB transaction flags
- Objects received by git-receive-pack(1) are quarantined in a temporary
- "incoming" directory and migrated into the object database prior to the
- reference updates. The quarantine is currently managed through
- tmp_objdir directly. In a pluggable ODB future, how exactly an object
- gets written to a transaction may vary for a given ODB source. Refactor
- git-receive-pack(1) to use the ODB transaction interfaces to manage the
- object staging area in a more agnostic manner accordingly.
+ The temporary directory used by git-receive-pack(1) to write objects is
+ managed slightly differently than how it is done via ODB transactions:
- Note that the temporary directory created for git-receive-pack(1) is
- eagerly created and uses a different prefix name. This behavior is
- special cased in the "files" backend by having odb_transaction_begin()
- callers that require this behavior provide an ODB_TRANSACTION_RECEIVE
- flag.
+ - The temporary directory is eagerly created upfront, instead of
+ waiting for the first object write.
+
+ - The prefix name of the temporary directory is "incoming" instead of
+ "bulk-fsync".
+
+ In a subsequent commit, git-receive-pack(1) will use ODB transactions
+ instead of tmp_objdir directly. To provide a means to configure the
+ same transaction behavior, introduce enum odb_transaction_flags and
+ the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB
+ transactions using the "files" backend to be set up for
+ git-receive-pack(1). Transaction call sites are updated accordingly to
+ provide the required flag parameter.
@@ builtin/add.c: int cmd_add(int argc,
ps_matched = xcalloc(pathspec.nr, 1);
if (add_renormalize)
- ## builtin/receive-pack.c ##
-@@ builtin/receive-pack.c: static enum {
- } use_keepalive;
- static int keepalive_in_sec = 5;
-
--static struct tmp_objdir *tmp_objdir;
--
- static struct proc_receive_ref {
- unsigned int want_add:1,
- want_delete:1,
-@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
- strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
- }
-
-- if (tmp_objdir)
-- strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
-+ if (the_repository->objects->transaction)
-+ strvec_pushv(&opt.env, odb_transaction_env(the_repository->objects->transaction));
-
- prepare_push_cert_sha1(&opt);
-
-@@ builtin/receive-pack.c: static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
- !delayed_reachability_test(si, i))
- oid_array_append(&extra, &si->shallow->oid[i]);
-
-- opt.env = tmp_objdir_env(tmp_objdir);
-+ opt.env = odb_transaction_env(the_repository->objects->transaction);
- setup_alternate_shallow(&shallow_lock, &opt.shallow_file, &extra);
- if (check_connected(command_singleton_iterator, cmd, &opt)) {
- rollback_shallow_file(the_repository, &shallow_lock);
-@@ builtin/receive-pack.c: static void set_connectivity_errors(struct command *commands,
- /* to be checked in update_shallow_ref() */
- continue;
-
-- opt.env = tmp_objdir_env(tmp_objdir);
-+ opt.env = odb_transaction_env(the_repository->objects->transaction);
- if (!check_connected(command_singleton_iterator, &singleton,
- &opt))
- continue;
-@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
- data.si = si;
- opt.err_fd = err_fd;
- opt.progress = err_fd && !quiet;
-- opt.env = tmp_objdir_env(tmp_objdir);
-+ opt.env = odb_transaction_env(the_repository->objects->transaction);
- opt.exclude_hidden_refs_section = "receive";
-
- if (check_connected(iterate_receive_command_list, &data, &opt))
-@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
- * Now we'll start writing out refs, which means the objects need
- * to be in their final positions so that other processes can see them.
- */
-- if (tmp_objdir_migrate(tmp_objdir) < 0) {
-+ if (odb_transaction_commit(the_repository->objects->transaction)) {
- for (cmd = commands; cmd; cmd = cmd->next) {
- if (!cmd->error_string)
- cmd->error_string = "unable to migrate objects to permanent storage";
- }
- return;
- }
-- tmp_objdir = NULL;
-
- check_aliased_updates(commands);
-
-@@ builtin/receive-pack.c: static void push_header_arg(struct strvec *args, struct pack_header *hdr)
- ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
- }
-
--static const char *unpack(int err_fd, struct shallow_info *si)
-+static const char *unpack(int err_fd, struct shallow_info *si,
-+ struct odb_transaction *transaction)
- {
- struct pack_header hdr;
- const char *hdr_err;
-@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
- strvec_push(&child.args, alt_shallow_file);
- }
-
-- tmp_objdir = tmp_objdir_create(the_repository, "incoming");
-- if (!tmp_objdir) {
-- if (err_fd > 0)
-- close(err_fd);
-- return "unable to create temporary object directory";
-- }
-- strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
--
-- /*
-- * Normally we just pass the tmp_objdir environment to the child
-- * processes that do the heavy lifting, but we may need to see these
-- * objects ourselves to set up shallow information.
-- */
-- tmp_objdir_add_as_alternate(tmp_objdir);
-+ strvec_pushv(&child.env, odb_transaction_env(transaction));
-
- if (ntohl(hdr.hdr_entries) < unpack_limit) {
- strvec_push(&child.args, "unpack-objects");
-@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
- return NULL;
- }
-
--static const char *unpack_with_sideband(struct shallow_info *si)
-+static const char *unpack_with_sideband(struct shallow_info *si,
-+ struct odb_transaction *transaction)
- {
- struct async muxer;
- const char *ret;
-
- if (!use_sideband)
-- return unpack(0, si);
-+ return unpack(0, si, transaction);
-
- use_keepalive = KEEPALIVE_AFTER_NUL;
- memset(&muxer, 0, sizeof(muxer));
-@@ builtin/receive-pack.c: static const char *unpack_with_sideband(struct shallow_info *si)
- if (start_async(&muxer))
- return NULL;
-
-- ret = unpack(muxer.in, si);
-+ ret = unpack(muxer.in, si, transaction);
-
- finish_async(&muxer);
- return ret;
-@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
- struct oid_array ref = OID_ARRAY_INIT;
- struct shallow_info si;
- struct packet_reader reader;
-+ struct odb_transaction *transaction = NULL;
-
- struct option options[] = {
- OPT__QUIET(&quiet, N_("quiet")),
-@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
- if (!si.nr_ours && !si.nr_theirs)
- shallow_update = 0;
- if (!delete_only(commands)) {
-- unpack_status = unpack_with_sideband(&si);
-+ if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
-+ unpack_status = "unable to start ODB transaction";
-+ else
-+ unpack_status = unpack_with_sideband(&si, transaction);
- update_shallow_info(commands, &si, &ref);
- }
- use_keepalive = KEEPALIVE_ALWAYS;
-
## builtin/unpack-objects.c ##
@@ builtin/unpack-objects.c: static void unpack_all(void)
progress = start_progress(the_repository,
@@ builtin/update-index.c: int cmd_update_index(int argc,
## cache-tree.c ##
@@ cache-tree.c: int cache_tree_update(struct index_state *istate, int flags)
-
trace_performance_enter();
trace2_region_enter("cache_tree", "update", istate->repo);
-- odb_transaction_begin_or_die(the_repository->objects, &transaction);
-+ odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
+ if (!inflight)
+- odb_transaction_begin_or_die(the_repository->objects, &transaction);
++ odb_transaction_begin_or_die(the_repository->objects, &transaction, 0);
i = update_one(istate->cache_tree, istate->cache, istate->cache_nr,
"", 0, &skip, flags);
- odb_transaction_commit(transaction);
+ if (!inflight)
## object-file.c ##
@@ object-file.c: struct odb_transaction_files {
@@ object-file.c: static int odb_transaction_files_prepare(struct odb_transaction *
return -1;
@@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
- struct object_database *odb = the_repository->objects;
- struct odb_transaction *transaction;
+ int inflight = !!transaction;
-- odb_transaction_begin_or_die(odb, &transaction);
-+ odb_transaction_begin_or_die(odb, &transaction, 0);
- ret = odb_transaction_write_object_stream(odb->transaction,
+ if (!inflight)
+- odb_transaction_begin_or_die(odb, &transaction);
++ odb_transaction_begin_or_die(odb, &transaction, 0);
+ ret = odb_transaction_write_object_stream(transaction,
&stream, xsize_t(st->st_size),
-@@ object-file.c: static const char **odb_transaction_files_env(struct odb_transaction *base) +@@ object-file.c: static int odb_transaction_files_env(struct odb_transaction *base,
}
int odb_transaction_files_begin(struct odb_source *source,
@@ object-file.c: static const char **odb_transaction_files_env(struct odb_transact
+ enum odb_transaction_flags flags)
{
struct odb_transaction_files *transaction;
- struct object_database *odb = source->odb;
+
@@ object-file.c: int odb_transaction_files_begin(struct odb_source *source,
transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream;
@@ object-file.c: int odb_transaction_files_begin(struct odb_source *source,
+ if (flags & ODB_TRANSACTION_RECEIVE) {
+ /*
+ * ODB transactions for git-receive-pack(1) eagerly create a
-+ * temporary directory and use a different prefix.
++ * temporary directory and use a different temporary directory
++ * prefix.
++ *
++ * NEEDSWORK: This transaction flag is only used by the "files"
++ * backend to special case temporary directory set up and
++ * handling. Ideally transaction users should not have to care
++ * though. To avoid this, we could eagerly create the temporary
++ * directory and use the same prefix name for all transactions.
+ */
+ transaction->prefix = "incoming";
+ if (odb_transaction_files_prepare(&transaction->base)) {
@@ object-file.h
/* The maximum size for an object header. */
#define MAX_HEADER_LEN 32
@@ object-file.h: struct odb_transaction;
- * pending, out is set to NULL.
+ * to make new objects visible.
*/
int odb_transaction_files_begin(struct odb_source *source,
- struct odb_transaction **out);
@@ odb/transaction.c
{
int ret;
-@@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
- return 0;
- }
+ if (odb->transaction)
+ return -1;
- ret = odb_source_begin_transaction(odb->sources, out);
+ ret = odb_source_begin_transaction(odb->sources, out, flags);
@@ odb/transaction.h
/*
* A transaction may be started for an object database prior to writing new
@@ odb/transaction.h: struct odb_transaction {
- const char **(*env)(struct odb_transaction *transaction);
+ int (*env)(struct odb_transaction *transaction, struct strvec *env);
};
++/* Flags used to configure an ODB transaction. */
+enum odb_transaction_flags {
++ /* Configures the transaction for use with git-receive-pack(1). */
+ ODB_TRANSACTION_RECEIVE = (1 << 0),
+};
+
@@ odb/transaction.h: struct odb_transaction {
* Starts an ODB transaction and returns it via out. Subsequent objects are
* written to the transaction and not committed until odb_transaction_commit()
@@ odb/transaction.h: struct odb_transaction {
- * error. If the ODB already has a pending transaction, out is set to NULL.
+ * ODB already has an inflight transaction pending.
*/
int odb_transaction_begin(struct object_database *odb,
- struct odb_transaction **out);
@@ odb/transaction.h: struct odb_transaction {
## read-cache.c ##
@@ read-cache.c: int add_files_to_cache(struct repository *repo, const char *prefix,
- * This function is invoked from commands other than 'add', which
* may not have their own transaction active.
*/
-- odb_transaction_begin_or_die(repo->objects, &transaction);
-+ odb_transaction_begin_or_die(repo->objects, &transaction, 0);
+ if (!inflight)
+- odb_transaction_begin_or_die(repo->objects, &transaction);
++ odb_transaction_begin_or_die(repo->objects, &transaction, 0);
run_diff_files(&rev, DIFF_RACY_IS_MODIFIED);
- odb_transaction_commit(transaction);
-
+ if (!inflight)
+ odb_transaction_commit(transaction);
-: ---------- > 10: 6c8d878349 builtin/receive-pack: drop redundant tmpdir env 4: e9303f9b08 ! 11: 8db95fef56 odb/transaction: propagate commit errors
@@ Metadata
Author: Justin Tobler <[email protected]>
## Commit message ##
- odb/transaction: propagate commit errors
+ builtin/receive-pack: stage incoming objects via ODB transactions
- When odb_transaction_commit() is invoked, the return value of the
- backend commit callback is silently discarded. A backend has no way
- to signal that committing failed, such as when the "files" backend
- cannot migrate its temporary object directory into the permanent
- ODB.
+ Objects received by git-receive-pack(1) are quarantined in a temporary
+ "incoming" directory and migrated into the object database prior to the
+ reference updates. The quarantine is currently managed through
+ tmp_objdir directly. In a pluggable ODB future, how exactly an object
+ gets written to a transaction may vary for a given ODB source. Refactor
+ git-receive-pack(1) to use the ODB transaction interfaces to manage the
+ object staging area in a more agnostic manner accordingly.
- In a subsequent commit, git-receive-pack(1) starts using ODB transaction
- to stage objects and consequently cares about such failures so it can
- handle the error appropriately. Change the commit callback signature to
- return an int error code and have odb_transaction_commit() forward it
- accordingly.
+ Note that the ODB transaction is now responsible for managing the
+ primary and alternate ODBs for the repository. One small change as a
+ result is that the temporary directory is now applied as the primary ODB
+ in the main process instead of an alternate. This does not change
+ anything for git-receive-pack(1) though because it only needs access to
+ the newly written objects and doesn't care how exactly it is set up.
- ## odb/transaction.c ##
-@@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
- return ret;
+ ## builtin/receive-pack.c ##
+@@
+ #include "sigchain.h"
+ #include "string-list.h"
+ #include "strvec.h"
+-#include "tmp-objdir.h"
+ #include "trace.h"
+ #include "trace2.h"
+ #include "version.h"
+@@ builtin/receive-pack.c: static enum {
+ } use_keepalive;
+ static int keepalive_in_sec = 5;
+
+-static struct tmp_objdir *tmp_objdir;
+-
+ static struct proc_receive_ref {
+ unsigned int want_add:1,
+ want_delete:1,
+@@ builtin/receive-pack.c: static void receive_hook_feed_state_free(void *data)
+ static int run_receive_hook(struct command *commands,
+ const char *hook_name,
+ int skip_broken,
++ struct odb_transaction *transaction,
+ const struct string_list *push_options)
+ {
+ struct run_hooks_opt opt = RUN_HOOKS_OPT_INIT;
+@@ builtin/receive-pack.c: static int run_receive_hook(struct command *commands,
+ strvec_push(&opt.env, "GIT_PUSH_OPTION_COUNT");
+ }
+
+- if (tmp_objdir)
+- strvec_pushv(&opt.env, tmp_objdir_env(tmp_objdir));
++ if (transaction)
++ odb_transaction_env(transaction, &opt.env);
+
+ prepare_push_cert_sha1(&opt);
+
+@@ builtin/receive-pack.c: static const struct object_id *command_singleton_iterator(void *cb_data)
}
--void odb_transaction_commit(struct odb_transaction *transaction) -+int odb_transaction_commit(struct odb_transaction *transaction)
+ static void set_connectivity_errors(struct command *commands,
+- struct shallow_info *si)
++ struct shallow_info *si,
++ struct odb_transaction *transaction)
{
-+ int ret;
+ struct command *cmd;
+
+ for (cmd = commands; cmd; cmd = cmd->next) {
+ struct command *singleton = cmd;
+ struct check_connected_options opt = CHECK_CONNECTED_INIT;
++ struct strvec env = STRVEC_INIT;
+
+ if (shallow_update && si->shallow_ref[cmd->index])
+ /* to be checked in update_shallow_ref() */
+ continue;
+
+- opt.env = tmp_objdir_env(tmp_objdir);
++ odb_transaction_env(transaction, &env);
++ opt.env = env.v;
+
- if (!transaction)
-- return;
-+ return 0;
+ if (!check_connected(command_singleton_iterator, &singleton,
+ &opt))
+ continue;
- /*
- * Ensure the transaction ending matches the pending transaction.
- */
- ASSERT(transaction == transaction->source->odb->transaction);
+ cmd->error_string = "missing necessary objects";
++
++ strvec_clear(&env);
+ }
+ }
+
+@@ builtin/receive-pack.c: static void execute_commands_atomic(struct command *commands,
+ static void execute_commands(struct command *commands,
+ const char *unpacker_error,
+ struct shallow_info *si,
++ struct odb_transaction *transaction,
+ const struct string_list *push_options)
+ {
+ struct check_connected_options opt = CHECK_CONNECTED_INIT;
+@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
+ }
-- transaction->commit(transaction);
-+ ret = transaction->commit(transaction);
-+ if (ret)
-+ return ret;
+ if (!skip_connectivity_check) {
++ struct strvec env = STRVEC_INIT;
+
- transaction->source->odb->transaction = NULL;
- free(transaction);
+ if (use_sideband) {
+ memset(&muxer, 0, sizeof(muxer));
+ muxer.proc = copy_to_sideband;
+@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
+ data.si = si;
+ opt.err_fd = err_fd;
+ opt.progress = err_fd && !quiet;
+- opt.env = tmp_objdir_env(tmp_objdir);
++ odb_transaction_env(transaction, &env);
++ opt.env = env.v;
+ opt.exclude_hidden_refs_section = "receive";
+
+ if (check_connected(iterate_receive_command_list, &data, &opt))
+- set_connectivity_errors(commands, si);
++ set_connectivity_errors(commands, si, transaction);
+
+ if (use_sideband)
+ finish_async(&muxer);
+
-+ return 0;
++ strvec_clear(&env);
+ }
+
+ reject_updates_to_hidden(commands);
+@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
+ }
+ }
+
+- if (run_receive_hook(commands, "pre-receive", 0, push_options)) {
++ if (run_receive_hook(commands, "pre-receive", 0, transaction, push_options)) {
+ for (cmd = commands; cmd; cmd = cmd->next) {
+ if (!cmd->error_string)
+ cmd->error_string = "pre-receive hook declined";
+@@ builtin/receive-pack.c: static void execute_commands(struct command *commands,
+ * Now we'll start writing out refs, which means the objects need
+ * to be in their final positions so that other processes can see them.
+ */
+- if (tmp_objdir_migrate(tmp_objdir) < 0) {
++ if (odb_transaction_commit(transaction)) {
+ for (cmd = commands; cmd; cmd = cmd->next) {
+ if (!cmd->error_string)
+ cmd->error_string = "unable to migrate objects to permanent storage";
+ }
+ return;
+ }
+- tmp_objdir = NULL;
+
+ check_aliased_updates(commands);
+
+@@ builtin/receive-pack.c: static void push_header_arg(struct strvec *args, struct pack_header *hdr)
+ ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries));
}
- int odb_transaction_write_object_stream(struct odb_transaction *transaction,
-
- ## odb/transaction.h ##
-@@ odb/transaction.h: static inline void odb_transaction_begin_or_die(struct object_database *odb,
- * Commits an ODB transaction making the written objects visible. If the
- * specified transaction is NULL, the function is a no-op.
- */
--void odb_transaction_commit(struct odb_transaction *transaction);
-+int odb_transaction_commit(struct odb_transaction *transaction);
-
- /*
- * Writes the object in the provided stream into the transaction. The resulting +-static const char *unpack(int err_fd, struct shallow_info *si)
++static const char *unpack(int err_fd, struct shallow_info *si,
++ struct odb_transaction *transaction)
+ {
+ struct pack_header hdr;
+ const char *hdr_err;
+@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
+ strvec_push(&child.args, alt_shallow_file);
+ }
+
+- tmp_objdir = tmp_objdir_create(the_repository, "incoming");
+- if (!tmp_objdir) {
+- if (err_fd > 0)
+- close(err_fd);
+- return "unable to create temporary object directory";
+- }
+- strvec_pushv(&child.env, tmp_objdir_env(tmp_objdir));
+-
+- /*
+- * Normally we just pass the tmp_objdir environment to the child
+- * processes that do the heavy lifting, but we may need to see these
+- * objects ourselves to set up shallow information.
+- */
+- tmp_objdir_add_as_alternate(tmp_objdir);
++ odb_transaction_env(transaction, &child.env);
+
+ if (ntohl(hdr.hdr_entries) < unpack_limit) {
+ strvec_push(&child.args, "unpack-objects");
+@@ builtin/receive-pack.c: static const char *unpack(int err_fd, struct shallow_info *si)
+ return NULL;
+ }
+
+-static const char *unpack_with_sideband(struct shallow_info *si)
++static const char *unpack_with_sideband(struct shallow_info *si,
++ struct odb_transaction *transaction)
+ {
+ struct async muxer;
+ const char *ret;
+
+ if (!use_sideband)
+- return unpack(0, si);
++ return unpack(0, si, transaction);
+
+ use_keepalive = KEEPALIVE_AFTER_NUL;
+ memset(&muxer, 0, sizeof(muxer));
+@@ builtin/receive-pack.c: static const char *unpack_with_sideband(struct shallow_info *si)
+ if (start_async(&muxer))
+ return NULL;
+
+- ret = unpack(muxer.in, si);
++ ret = unpack(muxer.in, si, transaction);
+
+ finish_async(&muxer);
+ return ret;
+@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
+ struct oid_array ref = OID_ARRAY_INIT;
+ struct shallow_info si;
+ struct packet_reader reader;
++ struct odb_transaction *transaction = NULL;
+
+ struct option options[] = {
+ OPT__QUIET(&quiet, N_("quiet")),
+@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
+ if (!si.nr_ours && !si.nr_theirs)
+ shallow_update = 0;
+ if (!delete_only(commands)) {
+- unpack_status = unpack_with_sideband(&si);
++ if (odb_transaction_begin(the_repository->objects, &transaction, ODB_TRANSACTION_RECEIVE))
++ unpack_status = "unable to start object transaction";
++ else
++ unpack_status = unpack_with_sideband(&si, transaction);
+ update_shallow_info(commands, &si, &ref);
+ }
+ use_keepalive = KEEPALIVE_ALWAYS;
+- execute_commands(commands, unpack_status, &si,
++ execute_commands(commands, unpack_status, &si, transaction,
+ &push_options);
+ delete_tempfile(&pack_lockfile);
+ sigchain_push(SIGPIPE, SIG_IGN);
+@@ builtin/receive-pack.c: int cmd_receive_pack(int argc,
+ else if (report_status)
+ report(commands, unpack_status);
+ sigchain_pop(SIGPIPE);
+- run_receive_hook(commands, "post-receive", 1,
++ run_receive_hook(commands, "post-receive", 1, NULL,
+ &push_options);
+ run_update_post_hook(commands);
+ free_commands(commands);
base-commit: ab776a62a78576513ee121424adb19597fbb7613
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v2 1/11 object-file: rename files transaction prepare function
The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.
Rename the function to odb_transaction_files_prepare() accordingly.
--- object-file.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
object-file.c +3 -3
@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; }; -static void prepare_loose_object_transaction (struct odb_transaction *base) +static void odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,static struct strbuf filename =STRBUF_INIT ;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -prepare_loose_object_transaction (loose->base.odb->transaction); +odb_transaction_files_prepare (loose->base.odb->transaction);odb_loose_path (loose, &filename, oid);
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,int hdrlen;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -prepare_loose_object_transaction (loose->base.odb->transaction); +odb_transaction_files_prepare (loose->base.odb->transaction);/* Since oid is not determined, save tmp file to odb path. */ strbuf_addf (&filename,"%s/" , loose->base.path);
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v2 2/11 object-file: rename files transaction fsync function
In reply to Justin ToblerWhen writing an object to a "files" ODB transaction, a full hardware flush is not initially performed during the fsync in fsync_loose_object_transaction() and instead delayed until the transaction is later committed.
To be more consistent with other "files" ODB transaction helpers, rename the function to odb_transaction_files_fsync() accordingly. The conditional in the helper is also slightly restructured to improve clarity to readers.
--- object-file.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
object-file.c +10 -6
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)tmp_objdir_replace_primary_odb (transaction->objdir,0 ); } -static void fsync_loose_object_transaction (struct odb_transaction *base, -int fd,const char *filename) +static void odb_transaction_files_fsync (struct odb_transaction *base, +int fd,const char *filename) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base); +if (!transaction || !transaction->objdir) { +fsync_or_die (fd, filename); +return ; + } +/* * If we have an active ODB transaction, we issue a call that * cleans the filesystem page cache but avoids a hardware flush
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,* before renaming the objects to their final names as part of * flush_batch_fsync. */ -if (!transaction || !transaction->objdir || -git_fsync (fd,FSYNC_WRITEOUT_ONLY ) <0 ) { +if (git_fsync (fd,FSYNC_WRITEOUT_ONLY ) <0 ) {if (errno ==ENOSYS )warning (_ ("core.fsyncMethod = batch is unsupported on this platform" ));fsync_or_die (fd, filename);
@@ -553,7 +557,7 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac/* * Issue a full hardware flush against a temporary file to ensure * that all objects are durable before any renames occur. The code in -* fsync_loose_object_transaction has already issued a writeout +* odb_transaction_files_fsync has already issued a writeout * request, but it has not flushed any writeback cache in the storage * hardware or any filesystem logs. This fsync call acts as a barrier * to ensure that the data in each new object file is durable before
@@ -582,7 +586,7 @@ static void close_loose_object(struct odb_source_loose *loose, goto out;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -fsync_loose_object_transaction (loose->base.odb->transaction, fd, filename); +odb_transaction_files_fsync (loose->base.odb->transaction, fd, filename);else if (fsync_object_files >0 )fsync_or_die (fd, filename);else
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v2 02/11] object-file: rename files transaction fsync function
Quoted 10 lines · Justin Tobler
object-file.c +0
-2@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)tmp_objdir_replace_primary_odb (transaction->objdir,0 ); } -static void fsync_loose_object_transaction (struct odb_transaction *base, -int fd,const char *filename)
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 530
Quoted 10 lines
object-file.c +6
-0+static void odb_transaction_files_fsync (struct odb_transaction *base, +int fd,const char *filename) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base); +if (!transaction || !transaction->objdir) { +fsync_or_die (fd, filename); +return ; + }
The change results in a tiny bit of duplication, but I agree that it's easier to reason about.
object-file.c static void fsync_loose_object_transaction(struct odb_transaction *base,
Quoted 1 line
object-file.c +0
-0@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,
Somewhat funny that the diff renderer decided to put the hunk header here instead of showing the single line that's now missing from the diff.
object-file.c static void fsync_loose_object_transaction(struct odb_transaction *base, · line 542
Quoted 9 lines
object-file.c +1
-2* before renaming the objects to their final names as part of * flush_batch_fsync. */ -if (!transaction || !transaction->objdir || -git_fsync (fd,FSYNC_WRITEOUT_ONLY ) <0 ) { +if (git_fsync (fd,FSYNC_WRITEOUT_ONLY ) <0 ) {if (errno ==ENOSYS )warning (_ ("core.fsyncMethod = batch is unsupported on this platform" ));fsync_or_die (fd, filename);
Thanks!
Patrick
PATCH v2 3/11 object-file: embed transaction flush logic in commit function
In reply to Justin ToblerWhen a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary. In a subsequent commit, the temporary directory is also used to write packfiles.
Instead of maintaining a separate helper function, embed the logic to flush and migrate the temporary directory directly into odb_transaction_files_commit().
--- object-file.c | 64 ++++++++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 36 deletions(-)
object-file.c +28 -36
@@ -543,41 +543,6 @@ static void odb_transaction_files_fsync(struct odb_transaction *base, } } -/* -* Cleanup after batch-mode fsync_object_files. -*/ -static void flush_loose_object_transaction (struct odb_transaction_files *transaction) -{ -struct strbuf temp_path =STRBUF_INIT ; -struct tempfile *temp; - -if (!transaction->objdir) -return ; - -/* -* Issue a full hardware flush against a temporary file to ensure -* that all objects are durable before any renames occur. The code in -* odb_transaction_files_fsync has already issued a writeout -* request, but it has not flushed any writeback cache in the storage -* hardware or any filesystem logs. This fsync call acts as a barrier -* to ensure that the data in each new object file is durable before -* the final name is visible. -*/ -strbuf_addf (&temp_path,"%s/bulk_fsync_XXXXXX" , -repo_get_object_directory (transaction->base.source->odb->repo)); - temp =xmks_tempfile (temp_path.buf); -fsync_or_die (get_tempfile_fd (temp),get_tempfile_path (temp)); -delete_tempfile (&temp); -strbuf_release (&temp_path); - -/* -* Make the object files visible in the primary ODB after their data is -* fully durable. -*/ -tmp_objdir_migrate (transaction->objdir); - transaction->objdir =NULL ; -} -/* Finalize a file on disk, and close it. */ static void close_loose_object (struct odb_source_loose *loose,int fd,const char *filename)
@@ -1679,7 +1644,34 @@ static void odb_transaction_files_commit(struct odb_transaction *base)struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base); -flush_loose_object_transaction (transaction); +if (transaction->objdir) { +struct strbuf temp_path =STRBUF_INIT ; +struct tempfile *temp; + +/* +* Issue a full hardware flush against a temporary file to ensure +* that all objects are durable before any renames occur. The code in +* odb_transaction_files_fsync has already issued a writeout +* request, but it has not flushed any writeback cache in the storage +* hardware or any filesystem logs. This fsync call acts as a barrier +* to ensure that the data in each new object file is durable before +* the final name is visible. +*/ +strbuf_addf (&temp_path,"%s/bulk_fsync_XXXXXX" , +repo_get_object_directory (transaction->base.source->odb->repo)); + temp =xmks_tempfile (temp_path.buf); +fsync_or_die (get_tempfile_fd (temp),get_tempfile_path (temp)); +delete_tempfile (&temp); +strbuf_release (&temp_path); + +/* +* Make the object files visible in the primary ODB after their data is +* fully durable. +*/ +tmp_objdir_migrate (transaction->objdir); + transaction->objdir =NULL ; + } +flush_packfile_transaction (transaction); }
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v2 03/11] object-file: embed transaction flush logic in commit function
Quoted 9 lines · Justin Tobler
When a "files" transaction is committed,
flush_loose_object_transaction()is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary. In a subsequent commit, the temporary directory is also used to write packfiles.Instead of maintaining a separate helper function, embed the logic to flush and migrate the temporary directory directly into
odb_transaction_files_commit().
The change itself looks simple enough, but this makes me wonder why we want to change this. Like, what subsequent step does this enable that would otherwise be harder to do?
Maybe this will be answered by a subsequent commit.
Patrick
Re: [PATCH v2 03/11] object-file: embed transaction flush logic in commit function
Quoted 7 lines · Patrick Steinhardt
Quoted 6 lines · Justin Tobler
When a "files" transaction is committed,
flush_loose_object_transaction()is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary. In a subsequent commit, the temporary directory is also used to write packfiles.
Quoted 7 lines
Quoted 3 lines
Instead of maintaining a separate helper function, embed the logic to flush and migrate the temporary directory directly into
odb_transaction_files_commit().The change itself looks simple enough, but this makes me wonder why we want to change this. Like, what subsequent step does this enable that would otherwise be harder to do?
Originally, I was planning to rename both {fsync,flush}_loose_object_transaction() to odb_transaction_files_{fsync,flush}() respectively. For the flush helper though, it's doing more than just "flushing" the temporary directory files; it's also migrating the objects to the primary ODB and reconfiguring the repository ODB sources accordingly.
This is really what you think of happening during the commit phase. Consequently, it made more sense to me organizationally to just embed the logic odb_transaction_files_commit().
Quoted 1 line
Maybe this will be answered by a subsequent commit.
Will update the commit message to properly explain this.
-Justin
PATCH v2 4/11 object-file: drop check for inflight transactions
In reply to Justin ToblerODB transactions are started via odb_transaction_begin() and contain validation to avoid starting multiple transactions at the same time. The "files" backend also has the same logic, but is redundant due to the generic layer already handling it. Drop this validation from the "files" backend accordingly.
--- object-file.c | 4 ---- object-file.h | 3 +-- 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -4
@@ -1678,10 +1678,6 @@ static void odb_transaction_files_commit(struct odb_transaction *base)struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) {struct odb_transaction_files *transaction; -struct object_database *odb = source->odb; - -if (odb->transaction) -return NULL ; transaction =xcalloc (1 ,sizeof (*transaction)); transaction->base.source = source;
object-file.h +1 -2
@@ -194,8 +194,7 @@ struct odb_transaction;/* * Tell the object database to optimize for adding * multiple objects. odb_transaction_files_commit must be called -* to make new objects visible. If a transaction is already -* pending, NULL is returned. +* to make new objects visible. */ struct odb_transaction *odb_transaction_files_begin (struct odb_source *source);
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v2 04/11] object-file: drop check for inflight transactions
Quoted 5 lines · Justin Tobler
ODB transactions are started via
odb_transaction_begin()and contain validation to avoid starting multiple transactions at the same time. The "files" backend also has the same logic, but is redundant due to the generic layer already handling it. Drop this validation from the "files" backend accordingly.
Makes sense, and it fixes a layering violation: in the best case, a source only has to care about itself and not about the owning object database. Managing object-database-level state should be done by the object database itself.
Patrick
PATCH v2 5/11 object-file: propagate files transaction errors
In reply to Justin ToblerThe "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle them as needed.
--- object-file.c | 26 ++++++++++++++++++-------- object-file.h | 3 ++- odb/source-files.c | 6 +----- odb/transaction.h | 7 +++++-- 4 files changed, 26 insertions(+), 16 deletions(-)
object-file.c +18 -8
@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; }; -static void odb_transaction_files_prepare (struct odb_transaction *base) +static int odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ; transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ; + +tmp_objdir_replace_primary_odb (transaction->objdir,0 ); + +return 0 ; } static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ -1639,7 +1643,7 @@ int read_loose_object(struct repository *repo,return ret; } -static void odb_transaction_files_commit (struct odb_transaction *base) +static int odb_transaction_files_commit (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base);
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)* Make the object files visible in the primary ODB after their data is * fully durable. */ -tmp_objdir_migrate (transaction->objdir); +if (tmp_objdir_migrate (transaction->objdir)) +return -1 ; + transaction->objdir =NULL ; }flush_packfile_transaction (transaction); + +return 0 ; } -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out) {struct odb_transaction_files *transaction;
@@ -1683,6 +1692,7 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source) transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + *out = &transaction->base; -return &transaction->base; +return 0 ; }
object-file.h +2 -1
@@ -196,6 +196,7 @@ struct odb_transaction;* multiple objects. odb_transaction_files_commit must be called * to make new objects visible. */ -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source); +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out);#endif /* OBJECT_FILE_H */
odb/source-files.c +1 -5
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,static int odb_source_files_begin_transaction (struct odb_source *source,struct odb_transaction **out) { -struct odb_transaction *tx =odb_transaction_files_begin (source); -if (!tx) -return -1 ; - *out = tx; -return 0 ; +return odb_transaction_files_begin (source, out); } static int odb_source_files_read_alternates(struct odb_source *source,
odb/transaction.h +5 -2
@@ -16,8 +16,11 @@ struct odb_transaction {/* The ODB source the transaction is opened against. */ struct odb_source *source; -/* The ODB source specific callback invoked to commit a transaction. */ -void (*commit)(struct odb_transaction *transaction); +/* +* The ODB source specific callback invoked to commit a transaction. +* Returns 0 on success, a negative error code otherwise. +*/ +int (*commit)(struct odb_transaction *transaction);/* * This callback is expected to write the given object stream into
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v2 05/11] object-file: propagate files transaction errors
Quoted 6 lines · Justin Tobler
object-file.c +0
-0@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin.
object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518
Quoted 10 lines
object-file.c +3
-3*/ if (!transaction || transaction->objdir) -return ; +return 0 ; transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ;
As far as I can see we don't report any errors as part of tmp_objdir_create(), so we should probably print an error here.
object-file.c static void odb_transaction_files_commit(struct odb_transaction *base) · line 1677
Quoted 8 lines
object-file.c +3
-1@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)* Make the object files visible in the primary ODB after their data is * fully durable. */ -tmp_objdir_migrate (transaction->objdir); +if (tmp_objdir_migrate (transaction->objdir)) +return -1 ; +
Likewise.
Patrick
Re: [PATCH v2 05/11] object-file: propagate files transaction errors
Quoted 21 lines · Patrick Steinhardt
Quoted 6 lines · Justin Tobler
object-file.c +0
-0@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. object-file.c static void odb_transaction_files_prepare(struct odb_transaction *base) · line 518
Quoted 10 lines
object-file.c +3
-3*/ if (!transaction || transaction->objdir) -return ; +return 0 ; transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return -1 ;As far as I can see we don't report any errors as part of
tmp_objdir_create(), so we should probably print an error here.
Quoted 10 lines
object-file.c static void odb_transaction_files_commit(struct odb_transaction *base) · line 1677
Quoted 8 lines
object-file.c +3
-1@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)* Make the object files visible in the primary ODB after their data is * fully durable. */ -tmp_objdir_migrate (transaction->objdir); +if (tmp_objdir_migrate (transaction->objdir)) +return -1 ; +Likewise.
Ya, printing some error messages seems like a good idea here. Will do so in the next version.
-Justin
PATCH v2 6/11 odb/transaction: propagate begin errors
In reply to Justin ToblerWhen odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.
In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.
Note that odb_transaction_begin() now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.
--- builtin/add.c | 2 +- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 7 +++++-- object-file.c | 10 +++++++--- odb/transaction.c | 12 ++++++++---- odb/transaction.h | 20 ++++++++++++++++---- read-cache.c | 7 +++++-- 8 files changed, 44 insertions(+), 18 deletions(-)
builtin/add.c +1 -1
@@ -581,7 +581,7 @@ int cmd_add(int argc,string_list_clear (&only_match_skip_worktree,0 ); } - transaction =odb_transaction_begin (repo->objects); +odb_transaction_begin_or_die (repo->objects, &transaction); ps_matched =xcalloc (pathspec.nr,1 );if (add_renormalize)
builtin/unpack-objects.c +1 -1
@@ -598,7 +598,7 @@ static void unpack_all(void) progress =start_progress (the_repository,_ ("Unpacking objects" ), nr_objects);CALLOC_ARRAY (obj_list, nr_objects); - transaction =odb_transaction_begin (the_repository->objects); +odb_transaction_begin_or_die (the_repository->objects, &transaction);for (i =0 ; i < nr_objects; i++) {unpack_one (i);display_progress (progress, i +1 );
builtin/update-index.c +1 -1
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,* Allow the object layer to optimize adding multiple objects in * a batch. */ - transaction =odb_transaction_begin (the_repository->objects); +odb_transaction_begin_or_die (the_repository->objects, &transaction);while (ctx.argc) {if (parseopt_state !=PARSE_OPT_DONE ) parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +5 -2
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,int cache_tree_update (struct index_state *istate,int flags) { +int inflight = !!the_repository->objects->transaction;struct odb_transaction *transaction;int skip, i;
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo); - transaction =odb_transaction_begin (the_repository->objects); +if (!inflight) +odb_transaction_begin_or_die (the_repository->objects, &transaction); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction);trace2_region_leave ("cache_tree" ,"update" , istate->repo);trace_performance_leave ("cache_tree_update" );if (i <0 )
object-file.c +7 -3
@@ -1354,13 +1354,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,if (flags &INDEX_WRITE_OBJECT ) {struct object_database *odb = the_repository->objects; -struct odb_transaction *transaction =odb_transaction_begin (odb); +struct odb_transaction *transaction = odb->transaction; +int inflight = !!transaction; - ret =odb_transaction_write_object_stream (odb->transaction, +if (!inflight) +odb_transaction_begin_or_die (odb, &transaction); + ret =odb_transaction_write_object_stream (transaction, &stream,xsize_t (st->st_size), oid); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction); }else { ret = hash_blob_stream(&stream, the_repository->hash_algo, oid,
odb/transaction.c +8 -4
@@ -2,14 +2,18 @@#include "odb/source.h" #include "odb/transaction.h" -struct odb_transaction *odb_transaction_begin (struct object_database *odb) +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out) { +int ret; +if (odb->transaction) -return NULL ; +return -1 ; -odb_source_begin_transaction (odb->sources, &odb->transaction); + ret =odb_source_begin_transaction (odb->sources, out); + odb->transaction = *out; -return odb->transaction; +return ret; }void odb_transaction_commit (struct odb_transaction *transaction)
odb/transaction.h +16 -4
@@ -1,6 +1,8 @@#ifndef ODB_TRANSACTION_H #define ODB_TRANSACTION_H +#include "git-compat-util.h" +#include "gettext.h" #include "odb.h" #include "odb/source.h"
@@ -36,11 +38,21 @@ struct odb_transaction { };/* -* Starts an ODB transaction. Subsequent objects are written to the transaction -* and not committed until odb_transaction_commit() is invoked on the -* transaction. If the ODB already has a pending transaction, NULL is returned. +* Starts an ODB transaction and returns it via `out`. Subsequent objects are +* written to the transaction and not committed until odb_transaction_commit() +* is invoked on the transaction. Returns 0 on success and a negative value on +* error. Note that it is considered an error to start a new transaction if the +* ODB already has an inflight transaction pending. */ -struct odb_transaction *odb_transaction_begin (struct object_database *odb); +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out); + +static inline void odb_transaction_begin_or_die (struct object_database *odb, +struct odb_transaction **out) +{ +if (odb_transaction_begin (odb, out)) +die (_ ("failed to start ODB transaction" )); +}/* * Commits an ODB transaction making the written objects visible. If the
read-cache.c +5 -2
@@ -4012,6 +4012,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix, const struct pathspec *pathspec, char *ps_matched, int include_sparse, int flags, int ignored_too ) { + int inflight = !!repo->objects->transaction;struct odb_transaction *transaction;struct update_callback_data data;struct rev_info rev;
@@ -4042,9 +4043,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,* This function is invoked from commands other than 'add', which * may not have their own transaction active. */ - transaction =odb_transaction_begin (repo->objects); +if (!inflight) +odb_transaction_begin_or_die (repo->objects, &transaction);run_diff_files (&rev,DIFF_RACY_IS_MODIFIED ); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction);release_revisions (&rev);return !!data.add_errors;
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v2 06/11] odb/transaction: propagate begin errors
Quoted 13 lines · Justin Tobler
When
odb_transaction_begin()is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor
odb_transaction_begin()to return an int error code and write the resulting transaction into an out parameter. Also introduceodb_transaction_begin_or_die()as a convenience for callsites that do not need to handle errors explicitly.
Quoted 4 lines
Note that
odb_transaction_begin()now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.
Yeah, this change is very much welcome and results in much saner behaviour with less surprises. Thanks for making the change.
Quoted 17 lines
cache-tree.c +1
-1@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,int cache_tree_update (struct index_state *istate,int flags) { +int inflight = !!the_repository->objects->transaction;struct odb_transaction *transaction;int skip, i;@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo); - transaction =odb_transaction_begin (the_repository->objects);
cache-tree.c int cache_tree_update(struct index_state *istate, int flags) · line 502
Quoted 10 lines
cache-tree.c +4
-1+if (!inflight) +odb_transaction_begin_or_die (the_repository->objects, &transaction); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction);trace2_region_leave ("cache_tree" ,"update" , istate->repo);trace_performance_leave ("cache_tree_update" );if (i <0 )
Callsites like this really make me wonder why we even care to create a transaction in the first place if we basically just commit it immediately anyway. And while it's a bit sad that we have so many sites where we don't really know whether we even have a transaction, I think it's a good change that we have now annotated them clearly. A subsequent patch series may then eventually refactor those sites so that we stop depending on odb->transaction and inject the transaction via a parameter.
odb/transaction.h line 4
Quoted 9 lines
odb/transaction.h +1
-0@@ -1,6 +1,8 @@#ifndef ODB_TRANSACTION_H #define ODB_TRANSACTION_H +#include "git-compat-util.h"
We typically don't include "git-compat-util.h" in header files.
Quoted 13 lines
odb/transaction.h +5
-3@@ -36,11 +38,21 @@ struct odb_transaction { };/* -* Starts an ODB transaction. Subsequent objects are written to the transaction -* and not committed until odb_transaction_commit() is invoked on the -* transaction. If the ODB already has a pending transaction, NULL is returned. +* Starts an ODB transaction and returns it via `out`. Subsequent objects are +* written to the transaction and not committed until odb_transaction_commit() +* is invoked on the transaction. Returns 0 on success and a negative value on +* error. Note that it is considered an error to start a new transaction if the +* ODB already has an inflight transaction pending. */
odb/transaction.h struct odb_transaction { · line 55
Quoted 10 lines
odb/transaction.h +9
-1-struct odb_transaction *odb_transaction_begin (struct object_database *odb); +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out); + +static inline void odb_transaction_begin_or_die (struct object_database *odb, +struct odb_transaction **out) +{ +if (odb_transaction_begin (odb, out)) +die (_ ("failed to start ODB transaction" )); +}
We could make it a bit simpler to use this function by continuing to return the transaction directly. But on the other hand this results in a more consistent interface.
Patrick
Re: [PATCH v2 06/11] odb/transaction: propagate begin errors
Quoted 17 lines · Patrick Steinhardt
Quoted 5 lines · Justin Tobler
cache-tree.c +0
-1@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo); - transaction =odb_transaction_begin (the_repository->objects);cache-tree.c int cache_tree_update(struct index_state *istate, int flags) · line 502
Quoted 10 lines
cache-tree.c +4
-1+if (!inflight) +odb_transaction_begin_or_die (the_repository->objects, &transaction); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction);trace2_region_leave ("cache_tree" ,"update" , istate->repo);trace_performance_leave ("cache_tree_update" );if (i <0 )
Quoted 8 lines
Callsites like this really make me wonder why we even care to create a transaction in the first place if we basically just commit it immediately anyway. And while it's a bit sad that we have so many sites where we don't really know whether we even have a transaction, I think it's a good change that we have now annotated them clearly. A subsequent patch series may then eventually refactor those sites so that we stop depending on
odb->transactionand inject the transaction via a parameter.
Call sites like the one mentioned above are using ODB transactions as an optimization to batch the full fsyncs in bulk. In cases where the is not already a transaction, they start one to take advantage of it.
I fully agree though that an ODB transaction should ideally be started at a higher layer and wired down to these call sites. I have a couple of patches in my tree that start to tackle this which I plan to send in another series. :)
Quoted 11 lines
odb/transaction.h line 4
Quoted 9 lines
odb/transaction.h +1
-0@@ -1,6 +1,8 @@#ifndef ODB_TRANSACTION_H #define ODB_TRANSACTION_H +#include "git-compat-util.h" We typically don't include "
git-compat-util.h" in header files.
Will remove. Thanks
Quoted 19 lines
Quoted 19 lines
odb/transaction.h +10
-4@@ -36,11 +38,21 @@ struct odb_transaction { };/* -* Starts an ODB transaction. Subsequent objects are written to the transaction -* and not committed until odb_transaction_commit() is invoked on the -* transaction. If the ODB already has a pending transaction, NULL is returned. +* Starts an ODB transaction and returns it via `out`. Subsequent objects are +* written to the transaction and not committed until odb_transaction_commit() +* is invoked on the transaction. Returns 0 on success and a negative value on +* error. Note that it is considered an error to start a new transaction if the +* ODB already has an inflight transaction pending. */ -struct odb_transaction *odb_transaction_begin (struct object_database *odb); +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out); + +static inline void odb_transaction_begin_or_die (struct object_database *odb, +struct odb_transaction **out)
Quoted 8 lines
odb/transaction.h struct odb_transaction { · line 55
Quoted 4 lines
odb/transaction.h +4
-0+{ +if (odb_transaction_begin (odb, out)) +die (_ ("failed to start ODB transaction" )); +}We could make it a bit simpler to use this function by continuing to return the transaction directly. But on the other hand this results in a more consistent interface.
Ya, I was a bit back and forth about this myself. I ultimately landed on keeping a more consistent interface though. Happy to change if others feel differently though.
-Justin
Re: [PATCH v2 06/11] odb/transaction: propagate begin errors
Quoted 27 lines · Justin Tobler
Quoted 17 lines · Patrick Steinhardt
Quoted 5 lines · Justin Tobler
cache-tree.c +0
-1@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo); - transaction =odb_transaction_begin (the_repository->objects);cache-tree.c int cache_tree_update(struct index_state *istate, int flags) · line 502
Quoted 10 lines
cache-tree.c +4
-1+if (!inflight) +odb_transaction_begin_or_die (the_repository->objects, &transaction); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction);trace2_region_leave ("cache_tree" ,"update" , istate->repo);trace_performance_leave ("cache_tree_update" );if (i <0 )Quoted 8 lines
Callsites like this really make me wonder why we even care to create a transaction in the first place if we basically just commit it immediately anyway. And while it's a bit sad that we have so many sites where we don't really know whether we even have a transaction, I think it's a good change that we have now annotated them clearly. A subsequent patch series may then eventually refactor those sites so that we stop depending on
odb->transactionand inject the transaction via a parameter.
Quoted 3 lines
Call sites like the one mentioned above are using ODB transactions as an optimization to batch the full fsyncs in bulk. In cases where the is not already a transaction, they start one to take advantage of it.
I know. But in the case where we don't want to batch we create the transaction anyway as far as I can see, and then we commit it immediately. So arguably, we could've just odb_write_object() and call it a day.
Quoted 4 lines
I fully agree though that an ODB transaction should ideally be started at a higher layer and wired down to these call sites. I have a couple of patches in my tree that start to tackle this which I plan to send in another series. :)
Yeah, let's not worry about that too much for now then. One step at a time :)
Quoted 28 lines
Quoted 19 lines
Quoted 19 lines
odb/transaction.h +10
-4@@ -36,11 +38,21 @@ struct odb_transaction { };/* -* Starts an ODB transaction. Subsequent objects are written to the transaction -* and not committed until odb_transaction_commit() is invoked on the -* transaction. If the ODB already has a pending transaction, NULL is returned. +* Starts an ODB transaction and returns it via `out`. Subsequent objects are +* written to the transaction and not committed until odb_transaction_commit() +* is invoked on the transaction. Returns 0 on success and a negative value on +* error. Note that it is considered an error to start a new transaction if the +* ODB already has an inflight transaction pending. */ -struct odb_transaction *odb_transaction_begin (struct object_database *odb); +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out); + +static inline void odb_transaction_begin_or_die (struct object_database *odb, +struct odb_transaction **out)Quoted 8 lines
odb/transaction.h struct odb_transaction { · line 55
Quoted 4 lines
odb/transaction.h +4
-0+{ +if (odb_transaction_begin (odb, out)) +die (_ ("failed to start ODB transaction" )); +}We could make it a bit simpler to use this function by continuing to return the transaction directly. But on the other hand this results in a more consistent interface.
Quoted 3 lines
Ya, I was a bit back and forth about this myself. I ultimately landed on keeping a more consistent interface though. Happy to change if others feel differently though.
As said, I can see both arguments. And ultimately, I don't care too much, so it's fine if this is just kept as-is.
Patrick
PATCH v2 7/11 odb/transaction: propagate commit errors
In reply to Justin ToblerWhen odb_transaction_commit() is invoked, the return value of the backend commit callback is silently discarded. A backend has no way to signal that committing failed, such as when the "files" backend cannot migrate its temporary object directory into the permanent ODB.
In a subsequent commit, git-receive-pack(1) starts using ODB transaction to stage objects and consequently cares about such failures so it can handle the error appropriately. Change the commit callback signature to return an int error code and have odb_transaction_commit() forward it accordingly.
--- odb/transaction.c | 13 ++++++++++--- odb/transaction.h | 7 ++++--- 2 files changed, 14 insertions(+), 6 deletions(-)
odb/transaction.c +10 -3
@@ -16,19 +16,26 @@ int odb_transaction_begin(struct object_database *odb,return ret; } -void odb_transaction_commit (struct odb_transaction *transaction) +int odb_transaction_commit (struct odb_transaction *transaction) { +int ret; +if (!transaction) -return ; +return 0 ;/* * Ensure the transaction ending matches the pending transaction. */ ASSERT (transaction == transaction->source->odb->transaction); - transaction->commit (transaction); + ret = transaction->commit (transaction); +if (ret) +return ret; + transaction->source->odb->transaction =NULL ;free (transaction); + +return 0 ; } int odb_transaction_write_object_stream(struct odb_transaction *transaction,
odb/transaction.h +4 -3
@@ -55,10 +55,11 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb, }/* -* Commits an ODB transaction making the written objects visible. If the -* specified transaction is NULL, the function is a no-op. +* Commits an ODB transaction making the written objects visible. Returns 0 on +* success, a negative error code otherwise. Note that, if the specified +* transaction is NULL, the function is a no-op and no error is returned. */ -void odb_transaction_commit (struct odb_transaction *transaction); +int odb_transaction_commit (struct odb_transaction *transaction);/* * Writes the object in the provided stream into the transaction. The resulting
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v2 07/11] odb/transaction: propagate commit errors
Quoted 22 lines · Justin Tobler
odb/transaction.c +4
-2@@ -16,19 +16,26 @@ int odb_transaction_begin(struct object_database *odb,return ret; } -void odb_transaction_commit (struct odb_transaction *transaction) +int odb_transaction_commit (struct odb_transaction *transaction) { +int ret; +if (!transaction) -return ; +return 0 ;/* * Ensure the transaction ending matches the pending transaction. */ ASSERT (transaction == transaction->source->odb->transaction);
odb/transaction.c int odb_transaction_begin(struct object_database *odb, · line 39
Quoted 10 lines
odb/transaction.c +6
-1- transaction->commit (transaction); + ret = transaction->commit (transaction); +if (ret) +return ret; + transaction->source->odb->transaction =NULL ;free (transaction); + +return 0 ; }
Doesn't this cause a leak now?
I think this interface here is doing the same mistake that our reference transactions did, where we automatically released the transaction on commit. That caused multiple lifetime issues with references all over the place.
This isn't an issue introduced by this patch series though, so it's fine to ignore this for now.
Patrick
Re: [PATCH v2 07/11] odb/transaction: propagate commit errors
Quoted 23 lines · Patrick Steinhardt
Quoted 22 lines · Justin Tobler
odb/transaction.c +4
-2@@ -16,19 +16,26 @@ int odb_transaction_begin(struct object_database *odb,return ret; } -void odb_transaction_commit (struct odb_transaction *transaction) +int odb_transaction_commit (struct odb_transaction *transaction) { +int ret; +if (!transaction) -return ; +return 0 ;/* * Ensure the transaction ending matches the pending transaction. */ ASSERT (transaction == transaction->source->odb->transaction);
Quoted 12 lines
odb/transaction.c int odb_transaction_begin(struct object_database *odb, · line 39
Quoted 10 lines
odb/transaction.c +6
-1- transaction->commit (transaction); + ret = transaction->commit (transaction); +if (ret) +return ret; + transaction->source->odb->transaction =NULL ;free (transaction); + +return 0 ; }Doesn't this cause a leak now?
Good call. Ya, if odb_transaction_commit() fails, we don't free the transaction. In the next version I'll go ahead and clear the transaction if we fail.
Quoted 4 lines
I think this interface here is doing the same mistake that our reference transactions did, where we automatically released the transaction on commit. That caused multiple lifetime issues with references all over the place.
Ya, it probaby makes sense to introduce a separate odb_transaction_release() function to make this explicit and update callers accordingly. I have another series I working on that introduces odb_transaction_abort(). This might be a good place to add it in too.
-Justin
PATCH v2 8/11 odb/transaction: add transaction env interface
In reply to Justin ToblerThe ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need access to uncommitted objects or write new objects in the staging area though.
Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction's staging area.
--- object-file.c | 14 ++++++++++++++ odb/transaction.c | 8 ++++++++ odb/transaction.h | 17 +++++++++++++++++ 3 files changed, 39 insertions(+)
object-file.c +14 -0
@@ -27,6 +27,7 @@#include "path.h" #include "read-cache-ll.h" #include "setup.h" +#include "strvec.h" #include "tempfile.h" #include "tmp-objdir.h"
@@ -1687,6 +1688,18 @@ static int odb_transaction_files_commit(struct odb_transaction *base)return 0 ; } +static int odb_transaction_files_env (struct odb_transaction *base, +struct strvec *env) +{ +struct odb_transaction_files *transaction = + container_of(base, struct odb_transaction_files, base); + +odb_transaction_files_prepare (&transaction->base); +strvec_pushv (env,tmp_objdir_env (transaction->objdir)); + +return 0 ; +} +int odb_transaction_files_begin (struct odb_source *source,struct odb_transaction **out) {
@@ -1696,6 +1709,7 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + transaction->base.env = odb_transaction_files_env; *out = &transaction->base;return 0 ;
odb/transaction.c +8 -0
@@ -44,3 +44,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction, {return transaction->write_object_stream (transaction, stream, len, oid); } + +int odb_transaction_env (struct odb_transaction *transaction,struct strvec *env) +{ +if (!transaction) +return 0 ; + +return transaction->env (transaction, env); +}
odb/transaction.h +17 -0
@@ -35,6 +35,14 @@ struct odb_transaction {int (*write_object_stream)(struct odb_transaction *transaction,struct odb_write_stream *stream,size_t len,struct object_id *oid); + +/* +* This callback is expected to populate the provided strvec with the +* environment variables that a child process should inherit so that its +* object writes participate in the transaction. Returns 0 on success, a +* negative error code otherwise. +*/ +int (*env)(struct odb_transaction *transaction,struct strvec *env); };/*
@@ -70,4 +78,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,struct odb_write_stream *stream, size_t len, struct object_id *oid); +/* +* Populates the provided strvec with the environment variables that a child +* process should inherit so that its object writes participate in the +* transaction, suitable for using via child_process.env. Returns 0 on success, +* a negative error code otherwise. Note that, if the specified transaction is +* NULL, the function is a no-op and no error is returned. +*/ +int odb_transaction_env (struct odb_transaction *transaction,struct strvec *env); +#endif
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v2 9/11 odb/transaction: introduce ODB transaction flags
In reply to Justin ToblerThe temporary directory used by git-receive-pack(1) to write objects is managed slightly differently than how it is done via ODB transactions:
- The temporary directory is eagerly created upfront, instead of waiting for the first object write.
- The prefix name of the temporary directory is "incoming" instead of
"bulk-fsync".
In a subsequent commit, git-receive-pack(1) will use ODB transactions instead of tmp_objdir directly. To provide a means to configure the same transaction behavior, introduce enum odb_transaction_flags and the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB transactions using the "files" backend to be set up for git-receive-pack(1). Transaction call sites are updated accordingly to provide the required flag parameter.
--- builtin/add.c | 2 +- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 2 +- object-file.c | 29 ++++++++++++++++++++++++++--- object-file.h | 4 +++- odb/source-files.c | 5 +++-- odb/source-inmemory.c | 3 ++- odb/source-loose.c | 3 ++- odb/source.h | 9 ++++++--- odb/transaction.c | 5 +++-- odb/transaction.h | 15 +++++++++++---- read-cache.c | 2 +- 13 files changed, 61 insertions(+), 22 deletions(-)
builtin/add.c +1 -1
@@ -581,7 +581,7 @@ int cmd_add(int argc,string_list_clear (&only_match_skip_worktree,0 ); } -odb_transaction_begin_or_die (repo->objects, &transaction); +odb_transaction_begin_or_die (repo->objects, &transaction,0 ); ps_matched =xcalloc (pathspec.nr,1 );if (add_renormalize)
builtin/unpack-objects.c +1 -1
@@ -598,7 +598,7 @@ static void unpack_all(void) progress =start_progress (the_repository,_ ("Unpacking objects" ), nr_objects);CALLOC_ARRAY (obj_list, nr_objects); -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 );for (i =0 ; i < nr_objects; i++) {unpack_one (i);display_progress (progress, i +1 );
builtin/update-index.c +1 -1
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,* Allow the object layer to optimize adding multiple objects in * a batch. */ -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 );while (ctx.argc) {if (parseopt_state !=PARSE_OPT_DONE ) parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
@@ -492,7 +492,7 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo);if (!inflight) -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 ); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags);if (!inflight)
object-file.c +26 -3
@@ -498,6 +498,7 @@ struct odb_transaction_files {struct tmp_objdir *objdir;struct transaction_packfile packfile; +const char *prefix; };static int odb_transaction_files_prepare (struct odb_transaction *base)
@@ -514,7 +515,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)if (!transaction || transaction->objdir)return 0 ; - transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); + transaction->objdir =tmp_objdir_create (base->source->odb->repo, transaction->prefix);if (!transaction->objdir)return -1 ;
@@ -1359,7 +1360,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,int inflight = !!transaction;if (!inflight) -odb_transaction_begin_or_die (odb, &transaction); +odb_transaction_begin_or_die (odb, &transaction,0 ); ret = odb_transaction_write_object_stream(transaction, &stream, xsize_t(st->st_size),
@@ -1701,7 +1702,8 @@ static int odb_transaction_files_env(struct odb_transaction *base, }int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {struct odb_transaction_files *transaction;
@@ -1710,6 +1712,27 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; transaction->base.env = odb_transaction_files_env; + + transaction->prefix ="bulk-fsync" ; +if (flags &ODB_TRANSACTION_RECEIVE ) { +/* +* ODB transactions for git-receive-pack(1) eagerly create a +* temporary directory and use a different temporary directory +* prefix. +* +* NEEDSWORK: This transaction flag is only used by the "files" +* backend to special case temporary directory set up and +* handling. Ideally transaction users should not have to care +* though. To avoid this, we could eagerly create the temporary +* directory and use the same prefix name for all transactions. +*/ + transaction->prefix ="incoming" ; +if (odb_transaction_files_prepare (&transaction->base)) { +free (transaction); +return -1 ; + } + } + *out = &transaction->base;return 0 ;
object-file.h +3 -1
@@ -5,6 +5,7 @@#include "object.h" #include "odb.h" #include "odb/source-loose.h" +#include "odb/transaction.h" /* The maximum size for an object header. */ #define MAX_HEADER_LEN 32
@@ -197,6 +198,7 @@ struct odb_transaction;* to make new objects visible. */ int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);#endif /* OBJECT_FILE_H */
odb/source-files.c +3 -2
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source, }static int odb_source_files_begin_transaction (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -return odb_transaction_files_begin (source, out); +return odb_transaction_files_begin (source, out, flags); } static int odb_source_files_read_alternates(struct odb_source *source,
odb/source-inmemory.c +2 -1
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source, } static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED, - struct odb_transaction **out UNUSED) + struct odb_transaction **out UNUSED, + enum odb_transaction_flags flags UNUSED) {return error ("in-memory source does not support transactions" ); }
odb/source-loose.c +2 -1
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source, } static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED, - struct odb_transaction **out UNUSED) + struct odb_transaction **out UNUSED, + enum odb_transaction_flags flags UNUSED) {/* TODO: this is a known omission that we'll want to address eventually. */ return error ("loose source does not support transactions" );
odb/source.h +6 -3
@@ -3,6 +3,7 @@#include "object.h" #include "odb.h" +#include "odb/transaction.h" enum odb_source_type {/*
@@ -228,7 +229,8 @@ struct odb_source {* negative error code otherwise. */ int (*begin_transaction)(struct odb_source *source, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);/* * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,* Returns 0 on success, a negative error code otherwise. */ static inline int odb_source_begin_transaction (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -return source->begin_transaction (source, out); +return source->begin_transaction (source, out, flags); }#endif
odb/transaction.c +3 -2
@@ -3,14 +3,15 @@#include "odb/transaction.h" int odb_transaction_begin (struct object_database *odb, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {int ret;if (odb->transaction)return -1 ; - ret =odb_source_begin_transaction (odb->sources, out); + ret =odb_source_begin_transaction (odb->sources, out, flags); odb->transaction = *out;return ret;
odb/transaction.h +11 -4
@@ -4,7 +4,6 @@#include "git-compat-util.h" #include "gettext.h" #include "odb.h" -#include "odb/source.h" /* * A transaction may be started for an object database prior to writing new
@@ -45,6 +44,12 @@ struct odb_transaction {int (*env)(struct odb_transaction *transaction,struct strvec *env); }; +/* Flags used to configure an ODB transaction. */ +enum odb_transaction_flags { +/* Configures the transaction for use with git-receive-pack(1). */ +ODB_TRANSACTION_RECEIVE = (1 <<0 ), +}; +/* * Starts an ODB transaction and returns it via `out`. Subsequent objects are * written to the transaction and not committed until odb_transaction_commit()
@@ -53,12 +58,14 @@ struct odb_transaction {* ODB already has an inflight transaction pending. */ int odb_transaction_begin (struct object_database *odb, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);static inline void odb_transaction_begin_or_die (struct object_database *odb, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -if (odb_transaction_begin (odb, out)) +if (odb_transaction_begin (odb, out, flags))die (_ ("failed to start ODB transaction" )); }
read-cache.c +1 -1
@@ -4044,7 +4044,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,* may not have their own transaction active. */ if (!inflight) -odb_transaction_begin_or_die (repo->objects, &transaction); +odb_transaction_begin_or_die (repo->objects, &transaction,0 );run_diff_files (&rev,DIFF_RACY_IS_MODIFIED );if (!inflight)odb_transaction_commit (transaction);
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v2 09/11] odb/transaction: introduce ODB transaction flags
Quoted 12 lines · Justin Tobler
object-file.c +4
-0@@ -1710,6 +1712,27 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; transaction->base.env = odb_transaction_files_env; + + transaction->prefix ="bulk-fsync" ; +if (flags &ODB_TRANSACTION_RECEIVE ) { +/*
object-file.c int odb_transaction_files_begin(struct odb_source *source, · line 1728
Quoted 10 lines
object-file.c +10
-0+* ODB transactions for git-receive-pack(1) eagerly create a +* temporary directory and use a different temporary directory +* prefix. +* +* NEEDSWORK: This transaction flag is only used by the "files" +* backend to special case temporary directory set up and +* handling. Ideally transaction users should not have to care +* though. To avoid this, we could eagerly create the temporary +* directory and use the same prefix name for all transactions. +*/
Yup, agreed, thanks for noting this here.
odb/transaction.h line 7
Quoted 9 lines
odb/transaction.h +0
-1@@ -4,7 +4,6 @@#include "git-compat-util.h" #include "gettext.h" #include "odb.h" -#include "odb/source.h"
This is curious, and likely a result of you adding "odb/transaction.h" to "odb/source.h".
odb/transaction.h struct odb_transaction { · line 55
Quoted 13 lines
odb/transaction.h +6
-0@@ -45,6 +44,12 @@ struct odb_transaction {int (*env)(struct odb_transaction *transaction,struct strvec *env); }; +/* Flags used to configure an ODB transaction. */ +enum odb_transaction_flags { +/* Configures the transaction for use with git-receive-pack(1). */ +ODB_TRANSACTION_RECEIVE = (1 <<0 ), +}; +/* * Starts an ODB transaction and returns it via `out`. Subsequent objects are * written to the transaction and not committed until odb_transaction_commit()
And this is the reason you have to add the include, so that the flags are visible in both "odb/source.h" and in "odb/transaction.h".
This makes me wonder whether there's really much value in having this header here be split out of "odb/source.h".
Patrick
Re: [PATCH v2 09/11] odb/transaction: introduce ODB transaction flags
Quoted 14 lines · Patrick Steinhardt
odb/transaction.h struct odb_transaction { · line 55
Quoted 9 lines · Justin Tobler
odb/transaction.h +6
-0+/* Flags used to configure an ODB transaction. */ +enum odb_transaction_flags { +/* Configures the transaction for use with git-receive-pack(1). */ +ODB_TRANSACTION_RECEIVE = (1 <<0 ), +}; +/* * Starts an ODB transaction and returns it via `out`. Subsequent objects are * written to the transaction and not committed until odb_transaction_commit() And this is the reason you have to add the include, so that the flags are visible in both "
odb/source.h" and in "odb/transaction.h".
Quoted 2 lines
This makes me wonder whether there's really much value in having this header here be split out of "
odb/source.h".
Ya, I've started wondering the same thing. A transaction implementation is always going to be tightly coupled to the ODB source it pertains too. It probably makes sense to merge "odb/transaction.{c,h}" with "odb/source.{c,h}". I'll leave it as-is for now and likely explore this is a future series though.
-Justin
PATCH v2 10/11 builtin/receive-pack: drop redundant tmpdir env
In reply to Justin ToblerWhen performing the connectivity checks for a shallow ref in update_shallow_ref(), the child process environment variables are populated via tmp_objdir_env(). This is unnecessary though as update_shallow_ref() is only reached after tmp_objdir_migrate() has been performed which means there is no longer a temporary directory that needs to be shared with child processes.
Drop the call to tmp_objdir_env() accordingly.
--- builtin/receive-pack.c | 1 - 1 file changed, 1 deletion(-)
builtin/receive-pack.c +0 -1
@@ -1363,7 +1363,6 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si) !delayed_reachability_test(si, i))oid_array_append (&extra, &si->shallow->oid[i]); - opt.env =tmp_objdir_env (tmp_objdir);setup_alternate_shallow (&shallow_lock, &opt.shallow_file, &extra);if (check_connected (command_singleton_iterator, cmd, &opt)) {rollback_shallow_file (the_repository, &shallow_lock);
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v2 10/11] builtin/receive-pack: drop redundant tmpdir env
Quoted 6 lines · Justin Tobler
When performing the connectivity checks for a shallow ref in
update_shallow_ref(), the child process environment variables are populated viatmp_objdir_env(). This is unnecessary though asupdate_shallow_ref()is only reached aftertmp_objdir_migrate()has been performed which means there is no longer a temporary directory that needs to be shared with child processes.
Right. We call it transitively via either execute_commands_atomic() or execute_commands_not_atomic(), both of which are called after tmp_objdir_migrate().
Patrick
PATCH v2 11/11 builtin/receive-pack: stage incoming objects via ODB transactions
In reply to Justin ToblerObjects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.
Note that the ODB transaction is now responsible for managing the primary and alternate ODBs for the repository. One small change as a result is that the temporary directory is now applied as the primary ODB in the main process instead of an alternate. This does not change anything for git-receive-pack(1) though because it only needs access to the newly written objects and doesn't care how exactly it is set up.
--- builtin/receive-pack.c | 68 ++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 33 deletions(-)
builtin/receive-pack.c +35 -33
@@ -37,7 +37,6 @@#include "sigchain.h" #include "string-list.h" #include "strvec.h" -#include "tmp-objdir.h" #include "trace.h" #include "trace2.h" #include "version.h"
@@ -112,8 +111,6 @@ static enum { } use_keepalive;static int keepalive_in_sec =5 ; -static struct tmp_objdir *tmp_objdir; - static struct proc_receive_ref { unsigned int want_add:1, want_delete:1,
@@ -926,6 +923,7 @@ static void receive_hook_feed_state_free(void *data)static int run_receive_hook (struct command *commands,const char *hook_name,int skip_broken, +struct odb_transaction *transaction,const struct string_list *push_options) {struct run_hooks_opt opt =RUN_HOOKS_OPT_INIT ;
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,strvec_push (&opt.env,"GIT_PUSH_OPTION_COUNT" ); } -if (tmp_objdir) -strvec_pushv (&opt.env,tmp_objdir_env (tmp_objdir)); +if (transaction) +odb_transaction_env (transaction, &opt.env);prepare_push_cert_sha1 (&opt);
@@ -1789,24 +1787,30 @@ static const struct object_id *command_singleton_iterator(void *cb_data) }static void set_connectivity_errors (struct command *commands, -struct shallow_info *si) +struct shallow_info *si, +struct odb_transaction *transaction) {struct command *cmd;for (cmd = commands; cmd; cmd = cmd->next) {struct command *singleton = cmd;struct check_connected_options opt =CHECK_CONNECTED_INIT ; +struct strvec env =STRVEC_INIT ;if (shallow_update && si->shallow_ref[cmd->index])/* to be checked in update_shallow_ref() */ continue ; - opt.env =tmp_objdir_env (tmp_objdir); +odb_transaction_env (transaction, &env); + opt.env = env.v; +if (!check_connected (command_singleton_iterator, &singleton, &opt))continue ; cmd->error_string ="missing necessary objects" ; + +strvec_clear (&env); } }
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,static void execute_commands (struct command *commands,const char *unpacker_error,struct shallow_info *si, +struct odb_transaction *transaction,const struct string_list *push_options) {struct check_connected_options opt =CHECK_CONNECTED_INIT ;
@@ -2043,6 +2048,8 @@ static void execute_commands(struct command *commands, }if (!skip_connectivity_check) { +struct strvec env =STRVEC_INIT ; +if (use_sideband) {memset (&muxer,0 ,sizeof (muxer)); muxer.proc = copy_to_sideband;
@@ -2056,14 +2063,17 @@ static void execute_commands(struct command *commands, data.si = si; opt.err_fd = err_fd; opt.progress = err_fd && !quiet; - opt.env =tmp_objdir_env (tmp_objdir); +odb_transaction_env (transaction, &env); + opt.env = env.v; opt.exclude_hidden_refs_section ="receive" ;if (check_connected (iterate_receive_command_list, &data, &opt)) -set_connectivity_errors (commands, si); +set_connectivity_errors (commands, si, transaction);if (use_sideband)finish_async (&muxer); + +strvec_clear (&env); }reject_updates_to_hidden (commands);
@@ -2084,7 +2094,7 @@ static void execute_commands(struct command *commands, } } -if (run_receive_hook (commands,"pre-receive" ,0 , push_options)) { +if (run_receive_hook (commands,"pre-receive" ,0 , transaction, push_options)) {for (cmd = commands; cmd; cmd = cmd->next) {if (!cmd->error_string) cmd->error_string ="pre-receive hook declined" ;
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,* Now we'll start writing out refs, which means the objects need * to be in their final positions so that other processes can see them. */ -if (tmp_objdir_migrate (tmp_objdir) <0 ) { +if (odb_transaction_commit (transaction)) {for (cmd = commands; cmd; cmd = cmd->next) {if (!cmd->error_string) cmd->error_string ="unable to migrate objects to permanent storage" ; }return ; } - tmp_objdir =NULL ;check_aliased_updates (commands);
@@ -2325,7 +2334,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr) ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries)); } -static const char *unpack (int err_fd,struct shallow_info *si) +static const char *unpack (int err_fd,struct shallow_info *si, +struct odb_transaction *transaction) {struct pack_header hdr;const char *hdr_err;
@@ -2350,20 +2360,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)strvec_push (&child.args, alt_shallow_file); } - tmp_objdir =tmp_objdir_create (the_repository,"incoming" ); -if (!tmp_objdir) { -if (err_fd >0 ) -close (err_fd); -return "unable to create temporary object directory" ; - } -strvec_pushv (&child.env,tmp_objdir_env (tmp_objdir)); - -/* -* Normally we just pass the tmp_objdir environment to the child -* processes that do the heavy lifting, but we may need to see these -* objects ourselves to set up shallow information. -*/ -tmp_objdir_add_as_alternate (tmp_objdir); +odb_transaction_env (transaction, &child.env);if (ntohl (hdr.hdr_entries) < unpack_limit) {strvec_push (&child.args,"unpack-objects" );
@@ -2430,13 +2427,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)return NULL ; } -static const char *unpack_with_sideband (struct shallow_info *si) +static const char *unpack_with_sideband (struct shallow_info *si, +struct odb_transaction *transaction) {struct async muxer;const char *ret;if (!use_sideband) -return unpack (0 , si); +return unpack (0 , si, transaction); use_keepalive =KEEPALIVE_AFTER_NUL ;memset (&muxer,0 ,sizeof (muxer));
@@ -2445,7 +2443,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)if (start_async (&muxer))return NULL ; - ret =unpack (muxer.in, si); + ret =unpack (muxer.in, si, transaction);finish_async (&muxer);return ret;
@@ -2622,6 +2620,7 @@ int cmd_receive_pack(int argc,struct oid_array ref =OID_ARRAY_INIT ;struct shallow_info si;struct packet_reader reader; +struct odb_transaction *transaction =NULL ;struct option options[] = {OPT__QUIET (&quiet,N_ ("quiet" )),
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,if (!si.nr_ours && !si.nr_theirs) shallow_update =0 ;if (!delete_only (commands)) { - unpack_status =unpack_with_sideband (&si); +if (odb_transaction_begin (the_repository->objects, &transaction,ODB_TRANSACTION_RECEIVE )) + unpack_status ="unable to start object transaction" ; +else + unpack_status =unpack_with_sideband (&si, transaction);update_shallow_info (commands, &si, &ref); } use_keepalive =KEEPALIVE_ALWAYS ; -execute_commands (commands, unpack_status, &si, +execute_commands (commands, unpack_status, &si, transaction, &push_options);delete_tempfile (&pack_lockfile);sigchain_push (SIGPIPE ,SIG_IGN );
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,else if (report_status)report (commands, unpack_status);sigchain_pop (SIGPIPE ); -run_receive_hook (commands,"post-receive" ,1 , +run_receive_hook (commands,"post-receive" ,1 ,NULL , &push_options);run_update_post_hook (commands);free_commands (commands);
Signature · 1 line
2.55.0.122.gf85a7e6620
Quoted 12 lines · Justin Tobler
Changes since V1:
- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling fromodb_transaction_files_begin().
-odb_transaction_begin()now returns an error if there is already an inflight transaction pending instead of setting theoutpointer to NULL.
- Updatedodb_transaction_env()to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child
Quoted 8 lines
processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it fromthe_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.
I've got a couple smaller nits, but overall I'm quite happy with the shape of this series now. Thanks!
Patrick
Quoted 22 lines · Patrick Steinhardt
Quoted 12 lines · Justin Tobler
Changes since V1:
- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling fromodb_transaction_files_begin().
-odb_transaction_begin()now returns an error if there is already an inflight transaction pending instead of setting theoutpointer to NULL.
- Updatedodb_transaction_env()to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for childQuoted 8 lines
processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it fromthe_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.
Quoted 2 lines
I've got a couple smaller nits, but overall I'm quite happy with the shape of this series now. Thanks!
Thanks for the review! I'll send another version later today. :)
-Justin
v3 cover receive-pack: use ODB transactions to stage object writes
In reply to Justin ToblerGreetings,
This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.
Changes since V2:
- Clarified commit log reasoning for embedding flush_loose_object_transaction() logic in commit function.
- Started printed some error messages on transaction errors.
- Removed include statement.
- Fixed transaction leak on odb_transaction_commit() error.
Changes since V1:
- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling from odb_transaction_files_begin().
- odb_transaction_begin() now returns an error if there is already an inflight transaction pending instead of setting the out pointer to NULL.
- Updated odb_transaction_env() to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it from the_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.
Thanks,
-Justin
Justin Tobler (11):
object-file: rename files transaction prepare function
object-file: rename files transaction fsync function
object-file: embed transaction flush logic in commit function
object-file: drop check for inflight transactions
object-file: propagate files transaction errors odb/transaction: propagate begin errors odb/transaction: propagate commit errors odb/transaction: add transaction env interface odb/transaction: introduce ODB transaction flags builtin/receive-pack: drop redundant tmpdir env builtin/receive-pack: stage incoming objects via ODB transactions
builtin/add.c | 2 +- builtin/receive-pack.c | 69 ++++++++--------- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 7 +- object-file.c | 159 +++++++++++++++++++++++++-------------- object-file.h | 8 +- odb/source-files.c | 9 +-- odb/source-inmemory.c | 3 +- odb/source-loose.c | 3 +- odb/source.h | 9 ++- odb/transaction.c | 32 ++++++-- odb/transaction.h | 59 ++++++++++++--- read-cache.c | 7 +- 14 files changed, 241 insertions(+), 130 deletions(-)
Range-diff against v2:
1: 9c14b219ad = 1: 9c14b219ad object-file: rename files transaction prepare function 2: 5703a9e93b = 2: 5703a9e93b object-file: rename files transaction fsync function 3: 4c37398ac8 ! 3: 76204847f2 object-file: embed transaction flush logic in commit function
@@ Commit message
When a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into
- the primary. In a subsequent commit, the temporary directory is also
- used to write packfiles.
+ the primary and configuring the repository ODB source accordingly. The
+ function name here is a bit misleading because the helper is doing a bit
+ more than just "flushing" the transaction contents. Also, in a
+ subsequent commit, the transaction temporary directory is used to stage
+ packfiles and not just loose objects anymore.
- Instead of maintaining a separate helper function, embed the logic to
- flush and migrate the temporary directory directly into
- odb_transaction_files_commit().
+ Lift the helper function logic directly into
+ odb_transaction_files_commit() to more accurately signal to readers
+ the operation being performed.
4: 623c6b02ea = 4: c97eb7763f object-file: drop check for inflight transactions
5: ca59176657 ! 5: 1f3a1f7714 object-file: propagate files transaction errors
@@ object-file.c: static void odb_transaction_files_prepare(struct odb_transaction
- if (transaction->objdir)
- tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+ if (!transaction->objdir)
-+ return -1;
++ return error(_("unable to create temporary object directory"));
+
+ tmp_objdir_replace_primary_odb(transaction->objdir, 0);
+
@@ object-file.c: static void odb_transaction_files_commit(struct odb_transaction *
*/
- tmp_objdir_migrate(transaction->objdir);
+ if (tmp_objdir_migrate(transaction->objdir))
-+ return -1;
++ return error(_("unable to migrate temporary objects"));
+
transaction->objdir = NULL;
}
6: 717a1ce9a7 ! 6: 09d13272d5 odb/transaction: propagate begin errors
@@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
## odb/transaction.c ##
@@
+ #include "git-compat-util.h"
++#include "gettext.h"
#include "odb/source.h"
#include "odb/transaction.h"
@@ odb/transaction.c
+
if (odb->transaction)
- return NULL;
-+ return -1;
++ return error(_("object database transaction already pending"));
- odb_source_begin_transaction(odb->sources, &odb->transaction);
+ ret = odb_source_begin_transaction(odb->sources, out);
@@ odb/transaction.h
#ifndef ODB_TRANSACTION_H
#define ODB_TRANSACTION_H
-+#include "git-compat-util.h"
+#include "gettext.h"
#include "odb.h"
#include "odb/source.h"
7: ff8e133965 ! 7: 12833d6773 odb/transaction: propagate commit errors
@@ odb/transaction.c: int odb_transaction_begin(struct object_database *odb,
- transaction->commit(transaction);
+ ret = transaction->commit(transaction);
-+ if (ret)
-+ return ret;
-+
transaction->source->odb->transaction = NULL;
free(transaction);
+
-+ return 0;
++ return ret;
}
int odb_transaction_write_object_stream(struct odb_transaction *transaction,
8: 264ba94b83 = 8: f2586f2f34 odb/transaction: add transaction env interface
9: 1e0a491ef2 ! 9: 9d082b5e47 odb/transaction: introduce ODB transaction flags
@@ object-file.c: static int odb_transaction_files_prepare(struct odb_transaction *
- transaction->objdir = tmp_objdir_create(base->source->odb->repo, "bulk-fsync");
+ transaction->objdir = tmp_objdir_create(base->source->odb->repo, transaction->prefix);
if (!transaction->objdir)
- return -1;
+ return error(_("unable to create temporary object directory"));
@@ object-file.c: int index_fd(struct index_state *istate, struct object_id *oid,
int inflight = !!transaction;
@@ odb/transaction.c
int ret;
if (odb->transaction)
- return -1;
+ return error(_("object database transaction already pending"));
- ret = odb_source_begin_transaction(odb->sources, out);
+ ret = odb_source_begin_transaction(odb->sources, out, flags);
@@ odb/transaction.c
## odb/transaction.h ##
@@
- #include "git-compat-util.h"
+
#include "gettext.h"
#include "odb.h"
-#include "odb/source.h"
10: 6c8d878349 = 10: e11d8a6676 builtin/receive-pack: drop redundant tmpdir env
11: 8db95fef56 = 11: fee57c2817 builtin/receive-pack: stage incoming objects via ODB transactionsbase-commit: ab776a62a78576513ee121424adb19597fbb7613
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v3 1/11 object-file: rename files transaction prepare function
The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.
Rename the function to odb_transaction_files_prepare() accordingly.
--- object-file.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
object-file.c +3 -3
@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; }; -static void prepare_loose_object_transaction (struct odb_transaction *base) +static void odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,static struct strbuf filename =STRBUF_INIT ;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -prepare_loose_object_transaction (loose->base.odb->transaction); +odb_transaction_files_prepare (loose->base.odb->transaction);odb_loose_path (loose, &filename, oid);
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,int hdrlen;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -prepare_loose_object_transaction (loose->base.odb->transaction); +odb_transaction_files_prepare (loose->base.odb->transaction);/* Since oid is not determined, save tmp file to odb path. */ strbuf_addf (&filename,"%s/" , loose->base.path);
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v3 2/11 object-file: rename files transaction fsync function
In reply to Justin ToblerWhen writing an object to a "files" ODB transaction, a full hardware flush is not initially performed during the fsync in fsync_loose_object_transaction() and instead delayed until the transaction is later committed.
To be more consistent with other "files" ODB transaction helpers, rename the function to odb_transaction_files_fsync() accordingly. The conditional in the helper is also slightly restructured to improve clarity to readers.
--- object-file.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
object-file.c +10 -6
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)tmp_objdir_replace_primary_odb (transaction->objdir,0 ); } -static void fsync_loose_object_transaction (struct odb_transaction *base, -int fd,const char *filename) +static void odb_transaction_files_fsync (struct odb_transaction *base, +int fd,const char *filename) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base); +if (!transaction || !transaction->objdir) { +fsync_or_die (fd, filename); +return ; + } +/* * If we have an active ODB transaction, we issue a call that * cleans the filesystem page cache but avoids a hardware flush
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,* before renaming the objects to their final names as part of * flush_batch_fsync. */ -if (!transaction || !transaction->objdir || -git_fsync (fd,FSYNC_WRITEOUT_ONLY ) <0 ) { +if (git_fsync (fd,FSYNC_WRITEOUT_ONLY ) <0 ) {if (errno ==ENOSYS )warning (_ ("core.fsyncMethod = batch is unsupported on this platform" ));fsync_or_die (fd, filename);
@@ -553,7 +557,7 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac/* * Issue a full hardware flush against a temporary file to ensure * that all objects are durable before any renames occur. The code in -* fsync_loose_object_transaction has already issued a writeout +* odb_transaction_files_fsync has already issued a writeout * request, but it has not flushed any writeback cache in the storage * hardware or any filesystem logs. This fsync call acts as a barrier * to ensure that the data in each new object file is durable before
@@ -582,7 +586,7 @@ static void close_loose_object(struct odb_source_loose *loose, goto out;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -fsync_loose_object_transaction (loose->base.odb->transaction, fd, filename); +odb_transaction_files_fsync (loose->base.odb->transaction, fd, filename);else if (fsync_object_files >0 )fsync_or_die (fd, filename);else
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v3 3/11 object-file: embed transaction flush logic in commit function
In reply to Justin ToblerWhen a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary and configuring the repository ODB source accordingly. The function name here is a bit misleading because the helper is doing a bit more than just "flushing" the transaction contents. Also, in a subsequent commit, the transaction temporary directory is used to stage packfiles and not just loose objects anymore.
Lift the helper function logic directly into odb_transaction_files_commit() to more accurately signal to readers the operation being performed.
--- object-file.c | 64 ++++++++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 36 deletions(-)
object-file.c +28 -36
@@ -543,41 +543,6 @@ static void odb_transaction_files_fsync(struct odb_transaction *base, } } -/* -* Cleanup after batch-mode fsync_object_files. -*/ -static void flush_loose_object_transaction (struct odb_transaction_files *transaction) -{ -struct strbuf temp_path =STRBUF_INIT ; -struct tempfile *temp; - -if (!transaction->objdir) -return ; - -/* -* Issue a full hardware flush against a temporary file to ensure -* that all objects are durable before any renames occur. The code in -* odb_transaction_files_fsync has already issued a writeout -* request, but it has not flushed any writeback cache in the storage -* hardware or any filesystem logs. This fsync call acts as a barrier -* to ensure that the data in each new object file is durable before -* the final name is visible. -*/ -strbuf_addf (&temp_path,"%s/bulk_fsync_XXXXXX" , -repo_get_object_directory (transaction->base.source->odb->repo)); - temp =xmks_tempfile (temp_path.buf); -fsync_or_die (get_tempfile_fd (temp),get_tempfile_path (temp)); -delete_tempfile (&temp); -strbuf_release (&temp_path); - -/* -* Make the object files visible in the primary ODB after their data is -* fully durable. -*/ -tmp_objdir_migrate (transaction->objdir); - transaction->objdir =NULL ; -} -/* Finalize a file on disk, and close it. */ static void close_loose_object (struct odb_source_loose *loose,int fd,const char *filename)
@@ -1679,7 +1644,34 @@ static void odb_transaction_files_commit(struct odb_transaction *base)struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base); -flush_loose_object_transaction (transaction); +if (transaction->objdir) { +struct strbuf temp_path =STRBUF_INIT ; +struct tempfile *temp; + +/* +* Issue a full hardware flush against a temporary file to ensure +* that all objects are durable before any renames occur. The code in +* odb_transaction_files_fsync has already issued a writeout +* request, but it has not flushed any writeback cache in the storage +* hardware or any filesystem logs. This fsync call acts as a barrier +* to ensure that the data in each new object file is durable before +* the final name is visible. +*/ +strbuf_addf (&temp_path,"%s/bulk_fsync_XXXXXX" , +repo_get_object_directory (transaction->base.source->odb->repo)); + temp =xmks_tempfile (temp_path.buf); +fsync_or_die (get_tempfile_fd (temp),get_tempfile_path (temp)); +delete_tempfile (&temp); +strbuf_release (&temp_path); + +/* +* Make the object files visible in the primary ODB after their data is +* fully durable. +*/ +tmp_objdir_migrate (transaction->objdir); + transaction->objdir =NULL ; + } +flush_packfile_transaction (transaction); }
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v3 03/11] object-file: embed transaction flush logic in commit function
Quoted 9 lines · Justin Tobler
When a "files" transaction is committed,
flush_loose_object_transaction()is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary and configuring the repository ODB source accordingly. The function name here is a bit misleading because the helper is doing a bit more than just "flushing" the transaction contents. Also, in a subsequent commit, the transaction temporary directory is used to stage packfiles and not just loose objects anymore.
Quoted 3 lines
Lift the helper function logic directly into
odb_transaction_files_commit()to more accurately signal to readers the operation being performed.
This line break makes my eyes bleed.
Patrick
PATCH v3 4/11 object-file: drop check for inflight transactions
In reply to Justin ToblerODB transactions are started via odb_transaction_begin() and contain validation to avoid starting multiple transactions at the same time. The "files" backend also has the same logic, but is redundant due to the generic layer already handling it. Drop this validation from the "files" backend accordingly.
--- object-file.c | 4 ---- object-file.h | 3 +-- 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -4
@@ -1678,10 +1678,6 @@ static void odb_transaction_files_commit(struct odb_transaction *base)struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) {struct odb_transaction_files *transaction; -struct object_database *odb = source->odb; - -if (odb->transaction) -return NULL ; transaction =xcalloc (1 ,sizeof (*transaction)); transaction->base.source = source;
object-file.h +1 -2
@@ -194,8 +194,7 @@ struct odb_transaction;/* * Tell the object database to optimize for adding * multiple objects. odb_transaction_files_commit must be called -* to make new objects visible. If a transaction is already -* pending, NULL is returned. +* to make new objects visible. */ struct odb_transaction *odb_transaction_files_begin (struct odb_source *source);
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v3 5/11 object-file: propagate files transaction errors
In reply to Justin ToblerThe "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle them as needed.
--- object-file.c | 26 ++++++++++++++++++-------- object-file.h | 3 ++- odb/source-files.c | 6 +----- odb/transaction.h | 7 +++++-- 4 files changed, 26 insertions(+), 16 deletions(-)
object-file.c +18 -8
@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; }; -static void odb_transaction_files_prepare (struct odb_transaction *base) +static int odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ; transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return error (_ ("unable to create temporary object directory" )); + +tmp_objdir_replace_primary_odb (transaction->objdir,0 ); + +return 0 ; } static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ -1639,7 +1643,7 @@ int read_loose_object(struct repository *repo,return ret; } -static void odb_transaction_files_commit (struct odb_transaction *base) +static int odb_transaction_files_commit (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base);
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)* Make the object files visible in the primary ODB after their data is * fully durable. */ -tmp_objdir_migrate (transaction->objdir); +if (tmp_objdir_migrate (transaction->objdir)) +return error (_ ("unable to migrate temporary objects" )); + transaction->objdir =NULL ; }flush_packfile_transaction (transaction); + +return 0 ; } -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out) {struct odb_transaction_files *transaction;
@@ -1683,6 +1692,7 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source) transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + *out = &transaction->base; -return &transaction->base; +return 0 ; }
object-file.h +2 -1
@@ -196,6 +196,7 @@ struct odb_transaction;* multiple objects. odb_transaction_files_commit must be called * to make new objects visible. */ -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source); +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out);#endif /* OBJECT_FILE_H */
odb/source-files.c +1 -5
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,static int odb_source_files_begin_transaction (struct odb_source *source,struct odb_transaction **out) { -struct odb_transaction *tx =odb_transaction_files_begin (source); -if (!tx) -return -1 ; - *out = tx; -return 0 ; +return odb_transaction_files_begin (source, out); } static int odb_source_files_read_alternates(struct odb_source *source,
odb/transaction.h +5 -2
@@ -16,8 +16,11 @@ struct odb_transaction {/* The ODB source the transaction is opened against. */ struct odb_source *source; -/* The ODB source specific callback invoked to commit a transaction. */ -void (*commit)(struct odb_transaction *transaction); +/* +* The ODB source specific callback invoked to commit a transaction. +* Returns 0 on success, a negative error code otherwise. +*/ +int (*commit)(struct odb_transaction *transaction);/* * This callback is expected to write the given object stream into
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v3 6/11 odb/transaction: propagate begin errors
In reply to Justin ToblerWhen odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.
In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.
Note that odb_transaction_begin() now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.
--- builtin/add.c | 2 +- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 7 +++++-- object-file.c | 10 +++++++--- odb/transaction.c | 13 +++++++++---- odb/transaction.h | 19 +++++++++++++++---- read-cache.c | 7 +++++-- 8 files changed, 44 insertions(+), 18 deletions(-)
builtin/add.c +1 -1
@@ -581,7 +581,7 @@ int cmd_add(int argc,string_list_clear (&only_match_skip_worktree,0 ); } - transaction =odb_transaction_begin (repo->objects); +odb_transaction_begin_or_die (repo->objects, &transaction); ps_matched =xcalloc (pathspec.nr,1 );if (add_renormalize)
builtin/unpack-objects.c +1 -1
@@ -598,7 +598,7 @@ static void unpack_all(void) progress =start_progress (the_repository,_ ("Unpacking objects" ), nr_objects);CALLOC_ARRAY (obj_list, nr_objects); - transaction =odb_transaction_begin (the_repository->objects); +odb_transaction_begin_or_die (the_repository->objects, &transaction);for (i =0 ; i < nr_objects; i++) {unpack_one (i);display_progress (progress, i +1 );
builtin/update-index.c +1 -1
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,* Allow the object layer to optimize adding multiple objects in * a batch. */ - transaction =odb_transaction_begin (the_repository->objects); +odb_transaction_begin_or_die (the_repository->objects, &transaction);while (ctx.argc) {if (parseopt_state !=PARSE_OPT_DONE ) parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +5 -2
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,int cache_tree_update (struct index_state *istate,int flags) { +int inflight = !!the_repository->objects->transaction;struct odb_transaction *transaction;int skip, i;
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo); - transaction =odb_transaction_begin (the_repository->objects); +if (!inflight) +odb_transaction_begin_or_die (the_repository->objects, &transaction); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction);trace2_region_leave ("cache_tree" ,"update" , istate->repo);trace_performance_leave ("cache_tree_update" );if (i <0 )
object-file.c +7 -3
@@ -1354,13 +1354,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,if (flags &INDEX_WRITE_OBJECT ) {struct object_database *odb = the_repository->objects; -struct odb_transaction *transaction =odb_transaction_begin (odb); +struct odb_transaction *transaction = odb->transaction; +int inflight = !!transaction; - ret =odb_transaction_write_object_stream (odb->transaction, +if (!inflight) +odb_transaction_begin_or_die (odb, &transaction); + ret =odb_transaction_write_object_stream (transaction, &stream,xsize_t (st->st_size), oid); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction); }else { ret = hash_blob_stream(&stream, the_repository->hash_algo, oid,
odb/transaction.c +9 -4
@@ -1,15 +1,20 @@#include "git-compat-util.h" +#include "gettext.h" #include "odb/source.h" #include "odb/transaction.h" -struct odb_transaction *odb_transaction_begin (struct object_database *odb) +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out) { +int ret; +if (odb->transaction) -return NULL ; +return error (_ ("object database transaction already pending" )); -odb_source_begin_transaction (odb->sources, &odb->transaction); + ret =odb_source_begin_transaction (odb->sources, out); + odb->transaction = *out; -return odb->transaction; +return ret; }void odb_transaction_commit (struct odb_transaction *transaction)
odb/transaction.h +15 -4
@@ -1,6 +1,7 @@#ifndef ODB_TRANSACTION_H #define ODB_TRANSACTION_H +#include "gettext.h" #include "odb.h" #include "odb/source.h"
@@ -36,11 +37,21 @@ struct odb_transaction { };/* -* Starts an ODB transaction. Subsequent objects are written to the transaction -* and not committed until odb_transaction_commit() is invoked on the -* transaction. If the ODB already has a pending transaction, NULL is returned. +* Starts an ODB transaction and returns it via `out`. Subsequent objects are +* written to the transaction and not committed until odb_transaction_commit() +* is invoked on the transaction. Returns 0 on success and a negative value on +* error. Note that it is considered an error to start a new transaction if the +* ODB already has an inflight transaction pending. */ -struct odb_transaction *odb_transaction_begin (struct object_database *odb); +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out); + +static inline void odb_transaction_begin_or_die (struct object_database *odb, +struct odb_transaction **out) +{ +if (odb_transaction_begin (odb, out)) +die (_ ("failed to start ODB transaction" )); +}/* * Commits an ODB transaction making the written objects visible. If the
read-cache.c +5 -2
@@ -4012,6 +4012,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix, const struct pathspec *pathspec, char *ps_matched, int include_sparse, int flags, int ignored_too ) { + int inflight = !!repo->objects->transaction;struct odb_transaction *transaction;struct update_callback_data data;struct rev_info rev;
@@ -4042,9 +4043,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,* This function is invoked from commands other than 'add', which * may not have their own transaction active. */ - transaction =odb_transaction_begin (repo->objects); +if (!inflight) +odb_transaction_begin_or_die (repo->objects, &transaction);run_diff_files (&rev,DIFF_RACY_IS_MODIFIED ); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction);release_revisions (&rev);return !!data.add_errors;
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v3 06/11] odb/transaction: propagate begin errors
Quoted 34 lines · Justin Tobler
When
odb_transaction_begin()is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor
odb_transaction_begin()to return an int error code and write the resulting transaction into an out parameter. Also introduceodb_transaction_begin_or_die()as a convenience for callsites that do not need to handle errors explicitly.Note that
odb_transaction_begin()now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.Signed-off-by Justin Tobler <[email protected]>---
...odb/transaction.c +3
-1@@ -1,15 +1,20 @@#include "git-compat-util.h" +#include "gettext.h" #include "odb/source.h" #include "odb/transaction.h" -struct odb_transaction *odb_transaction_begin (struct object_database *odb) +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out)
odb/transaction.c line 15
Quoted 10 lines
odb/transaction.c +5
-2{ +int ret; +if (odb->transaction) -return NULL ; +return error (_ ("object database transaction already pending" )); -odb_source_begin_transaction (odb->sources, &odb->transaction); + ret =odb_source_begin_transaction (odb->sources, out); + odb->transaction = *out;
Can odb_source_begin_transaction() ever fail? If so, and when it fails, would *out be left untouched? I am wondering if we want
if (!(ret = odb_source_begin_transaction(odb->sources, out)))
odb->transaction = *out;or something like that.
Re: [PATCH v3 06/11] odb/transaction: propagate begin errors
Quoted 43 lines · Junio C Hamano
Quoted 41 lines · Justin Tobler
When
odb_transaction_begin()is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor
odb_transaction_begin()to return an int error code and write the resulting transaction into an out parameter. Also introduceodb_transaction_begin_or_die()as a convenience for callsites that do not need to handle errors explicitly.Note that
odb_transaction_begin()now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.Signed-off-by Justin Tobler <[email protected]>---
...odb/transaction.c +6
-2@@ -1,15 +1,20 @@#include "git-compat-util.h" +#include "gettext.h" #include "odb/source.h" #include "odb/transaction.h" -struct odb_transaction *odb_transaction_begin (struct object_database *odb) +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out) { +int ret; +if (odb->transaction) -return NULL ; +return error (_ ("object database transaction already pending" ));
Quoted 6 lines
odb/transaction.c line 15
Quoted 3 lines
odb/transaction.c +2
-1-odb_source_begin_transaction (odb->sources, &odb->transaction); + ret =odb_source_begin_transaction (odb->sources, out); + odb->transaction = *out;Can
odb_source_begin_transaction()ever fail? If so, and when it fails, would *out be left untouched?
In this patch there it not yet a way for it to fail, but it can return an error later in the series. When an error is encountered though, *out _is_ left untouched.
Quoted 6 lines
I am wondering if we want
if (!(ret = odb_source_begin_transaction(odb->sources, out))) odb->transaction = *out;or something like that.
I think it is a good idea to for odb_transaction_begin() to ensure the repository transaction is only set on success though. Will update in the next version. Thanks
-Justin
PATCH v3 7/11 odb/transaction: propagate commit errors
In reply to Justin ToblerWhen odb_transaction_commit() is invoked, the return value of the backend commit callback is silently discarded. A backend has no way to signal that committing failed, such as when the "files" backend cannot migrate its temporary object directory into the permanent ODB.
In a subsequent commit, git-receive-pack(1) starts using ODB transaction to stage objects and consequently cares about such failures so it can handle the error appropriately. Change the commit callback signature to return an int error code and have odb_transaction_commit() forward it accordingly.
--- odb/transaction.c | 10 +++++++--- odb/transaction.h | 7 ++++--- 2 files changed, 11 insertions(+), 6 deletions(-)
odb/transaction.c +7 -3
@@ -17,19 +17,23 @@ int odb_transaction_begin(struct object_database *odb,return ret; } -void odb_transaction_commit (struct odb_transaction *transaction) +int odb_transaction_commit (struct odb_transaction *transaction) { +int ret; +if (!transaction) -return ; +return 0 ;/* * Ensure the transaction ending matches the pending transaction. */ ASSERT (transaction == transaction->source->odb->transaction); - transaction->commit (transaction); + ret = transaction->commit (transaction); transaction->source->odb->transaction =NULL ;free (transaction); + +return ret; } int odb_transaction_write_object_stream(struct odb_transaction *transaction,
odb/transaction.h +4 -3
@@ -54,10 +54,11 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb, }/* -* Commits an ODB transaction making the written objects visible. If the -* specified transaction is NULL, the function is a no-op. +* Commits an ODB transaction making the written objects visible. Returns 0 on +* success, a negative error code otherwise. Note that, if the specified +* transaction is NULL, the function is a no-op and no error is returned. */ -void odb_transaction_commit (struct odb_transaction *transaction); +int odb_transaction_commit (struct odb_transaction *transaction);/* * Writes the object in the provided stream into the transaction. The resulting
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v3 8/11 odb/transaction: add transaction env interface
In reply to Justin ToblerThe ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need access to uncommitted objects or write new objects in the staging area though.
Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction's staging area.
--- object-file.c | 14 ++++++++++++++ odb/transaction.c | 8 ++++++++ odb/transaction.h | 17 +++++++++++++++++ 3 files changed, 39 insertions(+)
object-file.c +14 -0
@@ -27,6 +27,7 @@#include "path.h" #include "read-cache-ll.h" #include "setup.h" +#include "strvec.h" #include "tempfile.h" #include "tmp-objdir.h"
@@ -1687,6 +1688,18 @@ static int odb_transaction_files_commit(struct odb_transaction *base)return 0 ; } +static int odb_transaction_files_env (struct odb_transaction *base, +struct strvec *env) +{ +struct odb_transaction_files *transaction = + container_of(base, struct odb_transaction_files, base); + +odb_transaction_files_prepare (&transaction->base); +strvec_pushv (env,tmp_objdir_env (transaction->objdir)); + +return 0 ; +} +int odb_transaction_files_begin (struct odb_source *source,struct odb_transaction **out) {
@@ -1696,6 +1709,7 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + transaction->base.env = odb_transaction_files_env; *out = &transaction->base;return 0 ;
odb/transaction.c +8 -0
@@ -42,3 +42,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction, {return transaction->write_object_stream (transaction, stream, len, oid); } + +int odb_transaction_env (struct odb_transaction *transaction,struct strvec *env) +{ +if (!transaction) +return 0 ; + +return transaction->env (transaction, env); +}
odb/transaction.h +17 -0
@@ -34,6 +34,14 @@ struct odb_transaction {int (*write_object_stream)(struct odb_transaction *transaction,struct odb_write_stream *stream,size_t len,struct object_id *oid); + +/* +* This callback is expected to populate the provided strvec with the +* environment variables that a child process should inherit so that its +* object writes participate in the transaction. Returns 0 on success, a +* negative error code otherwise. +*/ +int (*env)(struct odb_transaction *transaction,struct strvec *env); };/*
@@ -69,4 +77,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,struct odb_write_stream *stream, size_t len, struct object_id *oid); +/* +* Populates the provided strvec with the environment variables that a child +* process should inherit so that its object writes participate in the +* transaction, suitable for using via child_process.env. Returns 0 on success, +* a negative error code otherwise. Note that, if the specified transaction is +* NULL, the function is a no-op and no error is returned. +*/ +int odb_transaction_env (struct odb_transaction *transaction,struct strvec *env); +#endif
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v3 08/11] odb/transaction: add transaction env interface
object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1697
Quoted 7 lines · Justin Tobler
object-file.c +7
-0+static int odb_transaction_files_env (struct odb_transaction *base, +struct strvec *env) +{ +struct odb_transaction_files *transaction = + container_of(base, struct odb_transaction_files, base); + +odb_transaction_files_prepare (&transaction->base);
Can this fail? The caller of us would not notice that something went wrong, and ...
object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1698
Quoted 1 line
object-file.c +1
-0+strvec_pushv (env,tmp_objdir_env (transaction->objdir));
... happily ends up using transaction->objdir that may not be appropriate for it to use if it fails, no?
Quoted 2 lines
(quoted diff) +2
-0+ return 0; +}
Re: [PATCH v3 08/11] odb/transaction: add transaction env interface
Quoted 13 lines · Junio C Hamano
object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1697
Quoted 7 lines · Justin Tobler
object-file.c +7
-0+static int odb_transaction_files_env (struct odb_transaction *base, +struct strvec *env) +{ +struct odb_transaction_files *transaction = + container_of(base, struct odb_transaction_files, base); + +odb_transaction_files_prepare (&transaction->base);Can this fail? The caller of us would not notice that something went wrong, and ...
Quoted 4 lines
object-file.c static int odb_transaction_files_commit(struct odb_transaction *base) · line 1698
Quoted 1 line
object-file.c +1
-0+strvec_pushv (env,tmp_objdir_env (transaction->objdir));... happily ends up using
transaction->objdirthat may not be appropriate for it to use if it fails, no?
Ya, odb_transaction_files_prepare() can fail here. In practice, failure results in no temporary directory being created which tmp_objdir_env() does handle gracefully, but we should ideally still be reported the failure back to callers. Will update in the next version.
-Justin
PATCH v3 9/11 odb/transaction: introduce ODB transaction flags
In reply to Justin ToblerThe temporary directory used by git-receive-pack(1) to write objects is managed slightly differently than how it is done via ODB transactions:
- The temporary directory is eagerly created upfront, instead of waiting for the first object write.
- The prefix name of the temporary directory is "incoming" instead of
"bulk-fsync".
In a subsequent commit, git-receive-pack(1) will use ODB transactions instead of tmp_objdir directly. To provide a means to configure the same transaction behavior, introduce enum odb_transaction_flags and the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB transactions using the "files" backend to be set up for git-receive-pack(1). Transaction call sites are updated accordingly to provide the required flag parameter.
--- builtin/add.c | 2 +- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 2 +- object-file.c | 29 ++++++++++++++++++++++++++--- object-file.h | 4 +++- odb/source-files.c | 5 +++-- odb/source-inmemory.c | 3 ++- odb/source-loose.c | 3 ++- odb/source.h | 9 ++++++--- odb/transaction.c | 5 +++-- odb/transaction.h | 15 +++++++++++---- read-cache.c | 2 +- 13 files changed, 61 insertions(+), 22 deletions(-)
builtin/add.c +1 -1
@@ -581,7 +581,7 @@ int cmd_add(int argc,string_list_clear (&only_match_skip_worktree,0 ); } -odb_transaction_begin_or_die (repo->objects, &transaction); +odb_transaction_begin_or_die (repo->objects, &transaction,0 ); ps_matched =xcalloc (pathspec.nr,1 );if (add_renormalize)
builtin/unpack-objects.c +1 -1
@@ -598,7 +598,7 @@ static void unpack_all(void) progress =start_progress (the_repository,_ ("Unpacking objects" ), nr_objects);CALLOC_ARRAY (obj_list, nr_objects); -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 );for (i =0 ; i < nr_objects; i++) {unpack_one (i);display_progress (progress, i +1 );
builtin/update-index.c +1 -1
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,* Allow the object layer to optimize adding multiple objects in * a batch. */ -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 );while (ctx.argc) {if (parseopt_state !=PARSE_OPT_DONE ) parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
@@ -492,7 +492,7 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo);if (!inflight) -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 ); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags);if (!inflight)
object-file.c +26 -3
@@ -498,6 +498,7 @@ struct odb_transaction_files {struct tmp_objdir *objdir;struct transaction_packfile packfile; +const char *prefix; };static int odb_transaction_files_prepare (struct odb_transaction *base)
@@ -514,7 +515,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)if (!transaction || transaction->objdir)return 0 ; - transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); + transaction->objdir =tmp_objdir_create (base->source->odb->repo, transaction->prefix);if (!transaction->objdir)return error (_ ("unable to create temporary object directory" ));
@@ -1359,7 +1360,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,int inflight = !!transaction;if (!inflight) -odb_transaction_begin_or_die (odb, &transaction); +odb_transaction_begin_or_die (odb, &transaction,0 ); ret = odb_transaction_write_object_stream(transaction, &stream, xsize_t(st->st_size),
@@ -1701,7 +1702,8 @@ static int odb_transaction_files_env(struct odb_transaction *base, }int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {struct odb_transaction_files *transaction;
@@ -1710,6 +1712,27 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; transaction->base.env = odb_transaction_files_env; + + transaction->prefix ="bulk-fsync" ; +if (flags &ODB_TRANSACTION_RECEIVE ) { +/* +* ODB transactions for git-receive-pack(1) eagerly create a +* temporary directory and use a different temporary directory +* prefix. +* +* NEEDSWORK: This transaction flag is only used by the "files" +* backend to special case temporary directory set up and +* handling. Ideally transaction users should not have to care +* though. To avoid this, we could eagerly create the temporary +* directory and use the same prefix name for all transactions. +*/ + transaction->prefix ="incoming" ; +if (odb_transaction_files_prepare (&transaction->base)) { +free (transaction); +return -1 ; + } + } + *out = &transaction->base;return 0 ;
object-file.h +3 -1
@@ -5,6 +5,7 @@#include "object.h" #include "odb.h" #include "odb/source-loose.h" +#include "odb/transaction.h" /* The maximum size for an object header. */ #define MAX_HEADER_LEN 32
@@ -197,6 +198,7 @@ struct odb_transaction;* to make new objects visible. */ int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);#endif /* OBJECT_FILE_H */
odb/source-files.c +3 -2
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source, }static int odb_source_files_begin_transaction (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -return odb_transaction_files_begin (source, out); +return odb_transaction_files_begin (source, out, flags); } static int odb_source_files_read_alternates(struct odb_source *source,
odb/source-inmemory.c +2 -1
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source, } static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED, - struct odb_transaction **out UNUSED) + struct odb_transaction **out UNUSED, + enum odb_transaction_flags flags UNUSED) {return error ("in-memory source does not support transactions" ); }
odb/source-loose.c +2 -1
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source, } static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED, - struct odb_transaction **out UNUSED) + struct odb_transaction **out UNUSED, + enum odb_transaction_flags flags UNUSED) {/* TODO: this is a known omission that we'll want to address eventually. */ return error ("loose source does not support transactions" );
odb/source.h +6 -3
@@ -3,6 +3,7 @@#include "object.h" #include "odb.h" +#include "odb/transaction.h" enum odb_source_type {/*
@@ -228,7 +229,8 @@ struct odb_source {* negative error code otherwise. */ int (*begin_transaction)(struct odb_source *source, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);/* * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,* Returns 0 on success, a negative error code otherwise. */ static inline int odb_source_begin_transaction (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -return source->begin_transaction (source, out); +return source->begin_transaction (source, out, flags); }#endif
odb/transaction.c +3 -2
@@ -4,14 +4,15 @@#include "odb/transaction.h" int odb_transaction_begin (struct object_database *odb, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {int ret;if (odb->transaction)return error (_ ("object database transaction already pending" )); - ret =odb_source_begin_transaction (odb->sources, out); + ret =odb_source_begin_transaction (odb->sources, out, flags); odb->transaction = *out;return ret;
odb/transaction.h +11 -4
@@ -3,7 +3,6 @@#include "gettext.h" #include "odb.h" -#include "odb/source.h" /* * A transaction may be started for an object database prior to writing new
@@ -44,6 +43,12 @@ struct odb_transaction {int (*env)(struct odb_transaction *transaction,struct strvec *env); }; +/* Flags used to configure an ODB transaction. */ +enum odb_transaction_flags { +/* Configures the transaction for use with git-receive-pack(1). */ +ODB_TRANSACTION_RECEIVE = (1 <<0 ), +}; +/* * Starts an ODB transaction and returns it via `out`. Subsequent objects are * written to the transaction and not committed until odb_transaction_commit()
@@ -52,12 +57,14 @@ struct odb_transaction {* ODB already has an inflight transaction pending. */ int odb_transaction_begin (struct object_database *odb, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);static inline void odb_transaction_begin_or_die (struct object_database *odb, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -if (odb_transaction_begin (odb, out)) +if (odb_transaction_begin (odb, out, flags))die (_ ("failed to start ODB transaction" )); }
read-cache.c +1 -1
@@ -4044,7 +4044,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,* may not have their own transaction active. */ if (!inflight) -odb_transaction_begin_or_die (repo->objects, &transaction); +odb_transaction_begin_or_die (repo->objects, &transaction,0 );run_diff_files (&rev,DIFF_RACY_IS_MODIFIED );if (!inflight)odb_transaction_commit (transaction);
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v3 10/11 builtin/receive-pack: drop redundant tmpdir env
In reply to Justin ToblerWhen performing the connectivity checks for a shallow ref in update_shallow_ref(), the child process environment variables are populated via tmp_objdir_env(). This is unnecessary though as update_shallow_ref() is only reached after tmp_objdir_migrate() has been performed which means there is no longer a temporary directory that needs to be shared with child processes.
Drop the call to tmp_objdir_env() accordingly.
--- builtin/receive-pack.c | 1 - 1 file changed, 1 deletion(-)
builtin/receive-pack.c +0 -1
@@ -1363,7 +1363,6 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si) !delayed_reachability_test(si, i))oid_array_append (&extra, &si->shallow->oid[i]); - opt.env =tmp_objdir_env (tmp_objdir);setup_alternate_shallow (&shallow_lock, &opt.shallow_file, &extra);if (check_connected (command_singleton_iterator, cmd, &opt)) {rollback_shallow_file (the_repository, &shallow_lock);
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v3 11/11 builtin/receive-pack: stage incoming objects via ODB transactions
In reply to Justin ToblerObjects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.
Note that the ODB transaction is now responsible for managing the primary and alternate ODBs for the repository. One small change as a result is that the temporary directory is now applied as the primary ODB in the main process instead of an alternate. This does not change anything for git-receive-pack(1) though because it only needs access to the newly written objects and doesn't care how exactly it is set up.
--- builtin/receive-pack.c | 68 ++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 33 deletions(-)
builtin/receive-pack.c +35 -33
@@ -37,7 +37,6 @@#include "sigchain.h" #include "string-list.h" #include "strvec.h" -#include "tmp-objdir.h" #include "trace.h" #include "trace2.h" #include "version.h"
@@ -112,8 +111,6 @@ static enum { } use_keepalive;static int keepalive_in_sec =5 ; -static struct tmp_objdir *tmp_objdir; - static struct proc_receive_ref { unsigned int want_add:1, want_delete:1,
@@ -926,6 +923,7 @@ static void receive_hook_feed_state_free(void *data)static int run_receive_hook (struct command *commands,const char *hook_name,int skip_broken, +struct odb_transaction *transaction,const struct string_list *push_options) {struct run_hooks_opt opt =RUN_HOOKS_OPT_INIT ;
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,strvec_push (&opt.env,"GIT_PUSH_OPTION_COUNT" ); } -if (tmp_objdir) -strvec_pushv (&opt.env,tmp_objdir_env (tmp_objdir)); +if (transaction) +odb_transaction_env (transaction, &opt.env);prepare_push_cert_sha1 (&opt);
@@ -1789,24 +1787,30 @@ static const struct object_id *command_singleton_iterator(void *cb_data) }static void set_connectivity_errors (struct command *commands, -struct shallow_info *si) +struct shallow_info *si, +struct odb_transaction *transaction) {struct command *cmd;for (cmd = commands; cmd; cmd = cmd->next) {struct command *singleton = cmd;struct check_connected_options opt =CHECK_CONNECTED_INIT ; +struct strvec env =STRVEC_INIT ;if (shallow_update && si->shallow_ref[cmd->index])/* to be checked in update_shallow_ref() */ continue ; - opt.env =tmp_objdir_env (tmp_objdir); +odb_transaction_env (transaction, &env); + opt.env = env.v; +if (!check_connected (command_singleton_iterator, &singleton, &opt))continue ; cmd->error_string ="missing necessary objects" ; + +strvec_clear (&env); } }
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,static void execute_commands (struct command *commands,const char *unpacker_error,struct shallow_info *si, +struct odb_transaction *transaction,const struct string_list *push_options) {struct check_connected_options opt =CHECK_CONNECTED_INIT ;
@@ -2043,6 +2048,8 @@ static void execute_commands(struct command *commands, }if (!skip_connectivity_check) { +struct strvec env =STRVEC_INIT ; +if (use_sideband) {memset (&muxer,0 ,sizeof (muxer)); muxer.proc = copy_to_sideband;
@@ -2056,14 +2063,17 @@ static void execute_commands(struct command *commands, data.si = si; opt.err_fd = err_fd; opt.progress = err_fd && !quiet; - opt.env =tmp_objdir_env (tmp_objdir); +odb_transaction_env (transaction, &env); + opt.env = env.v; opt.exclude_hidden_refs_section ="receive" ;if (check_connected (iterate_receive_command_list, &data, &opt)) -set_connectivity_errors (commands, si); +set_connectivity_errors (commands, si, transaction);if (use_sideband)finish_async (&muxer); + +strvec_clear (&env); }reject_updates_to_hidden (commands);
@@ -2084,7 +2094,7 @@ static void execute_commands(struct command *commands, } } -if (run_receive_hook (commands,"pre-receive" ,0 , push_options)) { +if (run_receive_hook (commands,"pre-receive" ,0 , transaction, push_options)) {for (cmd = commands; cmd; cmd = cmd->next) {if (!cmd->error_string) cmd->error_string ="pre-receive hook declined" ;
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,* Now we'll start writing out refs, which means the objects need * to be in their final positions so that other processes can see them. */ -if (tmp_objdir_migrate (tmp_objdir) <0 ) { +if (odb_transaction_commit (transaction)) {for (cmd = commands; cmd; cmd = cmd->next) {if (!cmd->error_string) cmd->error_string ="unable to migrate objects to permanent storage" ; }return ; } - tmp_objdir =NULL ;check_aliased_updates (commands);
@@ -2325,7 +2334,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr) ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries)); } -static const char *unpack (int err_fd,struct shallow_info *si) +static const char *unpack (int err_fd,struct shallow_info *si, +struct odb_transaction *transaction) {struct pack_header hdr;const char *hdr_err;
@@ -2350,20 +2360,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)strvec_push (&child.args, alt_shallow_file); } - tmp_objdir =tmp_objdir_create (the_repository,"incoming" ); -if (!tmp_objdir) { -if (err_fd >0 ) -close (err_fd); -return "unable to create temporary object directory" ; - } -strvec_pushv (&child.env,tmp_objdir_env (tmp_objdir)); - -/* -* Normally we just pass the tmp_objdir environment to the child -* processes that do the heavy lifting, but we may need to see these -* objects ourselves to set up shallow information. -*/ -tmp_objdir_add_as_alternate (tmp_objdir); +odb_transaction_env (transaction, &child.env);if (ntohl (hdr.hdr_entries) < unpack_limit) {strvec_push (&child.args,"unpack-objects" );
@@ -2430,13 +2427,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)return NULL ; } -static const char *unpack_with_sideband (struct shallow_info *si) +static const char *unpack_with_sideband (struct shallow_info *si, +struct odb_transaction *transaction) {struct async muxer;const char *ret;if (!use_sideband) -return unpack (0 , si); +return unpack (0 , si, transaction); use_keepalive =KEEPALIVE_AFTER_NUL ;memset (&muxer,0 ,sizeof (muxer));
@@ -2445,7 +2443,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)if (start_async (&muxer))return NULL ; - ret =unpack (muxer.in, si); + ret =unpack (muxer.in, si, transaction);finish_async (&muxer);return ret;
@@ -2622,6 +2620,7 @@ int cmd_receive_pack(int argc,struct oid_array ref =OID_ARRAY_INIT ;struct shallow_info si;struct packet_reader reader; +struct odb_transaction *transaction =NULL ;struct option options[] = {OPT__QUIET (&quiet,N_ ("quiet" )),
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,if (!si.nr_ours && !si.nr_theirs) shallow_update =0 ;if (!delete_only (commands)) { - unpack_status =unpack_with_sideband (&si); +if (odb_transaction_begin (the_repository->objects, &transaction,ODB_TRANSACTION_RECEIVE )) + unpack_status ="unable to start object transaction" ; +else + unpack_status =unpack_with_sideband (&si, transaction);update_shallow_info (commands, &si, &ref); } use_keepalive =KEEPALIVE_ALWAYS ; -execute_commands (commands, unpack_status, &si, +execute_commands (commands, unpack_status, &si, transaction, &push_options);delete_tempfile (&pack_lockfile);sigchain_push (SIGPIPE ,SIG_IGN );
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,else if (report_status)report (commands, unpack_status);sigchain_pop (SIGPIPE ); -run_receive_hook (commands,"post-receive" ,1 , +run_receive_hook (commands,"post-receive" ,1 ,NULL , &push_options);run_update_post_hook (commands);free_commands (commands);
Signature · 1 line
2.55.0.122.gf85a7e6620
Re: [PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions
builtin/receive-pack.c static void execute_commands_atomic(struct command *commands, · line 2037
Quoted 9 lines · Justin Tobler
builtin/receive-pack.c +1
-0@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,static void execute_commands (struct command *commands,const char *unpacker_error,struct shallow_info *si, +struct odb_transaction *transaction,const struct string_list *push_options) {struct check_connected_options opt =CHECK_CONNECTED_INIT ;
Hidden in the context early in this function is an error return. When unpacker_error string is non NULL, we mark all the commands in the linked commands list as failed, and return early from this function.
Quoted 5 lines
builtin/receive-pack.c +0
-1@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,* Now we'll start writing out refs, which means the objects need * to be in their final positions so that other processes can see them. */ -if (tmp_objdir_migrate (tmp_objdir) <0 ) {
builtin/receive-pack.c static void execute_commands(struct command *commands, · line 2126
Quoted 10 lines
builtin/receive-pack.c +1
-1+if (odb_transaction_commit (transaction)) {for (cmd = commands; cmd; cmd = cmd->next) {if (!cmd->error_string) cmd->error_string ="unable to migrate objects to permanent storage" ; }return ; } - tmp_objdir =NULL ;check_aliased_updates (commands);
In the "happy case", execute_commands() would commit the transaction before going on to do the execute_commands_{atomic,nonatomic}() that appears later in it.
builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2708
Quoted 6 lines
builtin/receive-pack.c +1
-1@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,if (!si.nr_ours && !si.nr_theirs) shallow_update =0 ;if (!delete_only (commands)) { - unpack_status =unpack_with_sideband (&si); +if (odb_transaction_begin (the_repository->objects, &transaction,ODB_TRANSACTION_RECEIVE ))
In the "main" program, we start a transaction here, and
builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2711
Quoted 3 lines
builtin/receive-pack.c +3
-0+ unpack_status ="unable to start object transaction" ; + else + unpack_status =unpack_with_sideband (&si, transaction);
then call unpack_with_sideband(). It may fail.
builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2716
Quoted 6 lines
builtin/receive-pack.c +1
-1update_shallow_info (commands, &si, &ref); } use_keepalive =KEEPALIVE_ALWAYS ; -execute_commands (commands, unpack_status, &si, +execute_commands (commands, unpack_status, &si, transaction, &push_options);
And in such a case, execute_commands() returns without committing the transaction. Is there a need to add and make an odb_transaction_abort() call or something in such a case? Everything should be cleaned up upon process exit, and on file based backends, we probably let the tempfile/lockfile API do their thing to clean up, but are there other things we may want to clean up?
Quoted 11 lines
builtin/receive-pack.c +1
-1delete_tempfile (&pack_lockfile);sigchain_push (SIGPIPE ,SIG_IGN );@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,else if (report_status)report (commands, unpack_status);sigchain_pop (SIGPIPE ); -run_receive_hook (commands,"post-receive" ,1 , +run_receive_hook (commands,"post-receive" ,1 ,NULL , &push_options);run_update_post_hook (commands);free_commands (commands);
Re: [PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions
Quoted 14 lines · Junio C Hamano
builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2716
Quoted 6 lines · Justin Tobler
builtin/receive-pack.c +1
-1update_shallow_info (commands, &si, &ref); } use_keepalive =KEEPALIVE_ALWAYS ; -execute_commands (commands, unpack_status, &si, +execute_commands (commands, unpack_status, &si, transaction, &push_options);And in such a case,
execute_commands()returns without committing the transaction. Is there a need to add and make anodb_transaction_abort()call or something in such a case? Everything should be cleaned up upon process exit, and on file based backends, we probably let the tempfile/lockfile API do their thing to clean up, but are there other things we may want to clean up?
As you mentioned, if we exit before committing the ODB transaction, the temporary directory will get cleaned up when the process exits. I don't think there is anything else we need to cleanup that wouldn't be handled at exit though. Regardless, I do plan to add odb_transaction_abort() in a followup series and I think it would be nice to have an explicit "abort" here when we know that we are not going to commit anyways. I would like to defer this to my next series though.
-Justin
Re: [PATCH v3 11/11] builtin/receive-pack: stage incoming objects via ODB transactions
Quoted 16 lines · Justin Tobler
Quoted 14 lines · Junio C Hamano
builtin/receive-pack.c int cmd_receive_pack(int argc, · line 2716
Quoted 6 lines · Justin Tobler
builtin/receive-pack.c +1
-1update_shallow_info (commands, &si, &ref); } use_keepalive =KEEPALIVE_ALWAYS ; -execute_commands (commands, unpack_status, &si, +execute_commands (commands, unpack_status, &si, transaction, &push_options);And in such a case,
execute_commands()returns without committing the transaction. Is there a need to add and make anodb_transaction_abort()call or something in such a case? Everything should be cleaned up upon process exit, and on file based backends, we probably let the tempfile/lockfile API do their thing to clean up, but are there other things we may want to clean up?
Quoted 7 lines
As you mentioned, if we exit before committing the ODB transaction, the temporary directory will get cleaned up when the process exits. I don't think there is anything else we need to cleanup that wouldn't be handled at exit though. Regardless, I do plan to add
odb_transaction_abort()in a followup series and I think it would be nice to have an explicit "abort" here when we know that we are not going to commit anyways. I would like to defer this to my next series though.
Sounds good. We cannot trigger receive-pack as a subroutine call in a long running daemon until that happens, but that is OK for now. One step at a time.
Quoted 6 lines · Justin Tobler
Changes since V2:
- Clarified commit log reasoning for embeddingflush_loose_object_transaction()logic in commit function.
- Started printed some error messages on transaction errors.
- Removed include statement.
- Fixed transaction leak onodb_transaction_commit()error.
Thanks, the changes all look good to me and I don't have anything else to add.
Patrick
v4 cover receive-pack: use ODB transactions to stage object writes
In reply to Justin ToblerGreetings,
This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.
Changes since V3:
- Removed ugly line break in commit message to prevent eye strain.
- odb_transaction_begin() now only sets the repository transaction on success.
- odb_transaction_env() now bubbles up error when failing to create the temporary directory.
Changes since V2:
- Clarified commit log reasoning for embedding flush_loose_object_transaction() logic in commit function.
- Started printed some error messages on transaction errors.
- Removed include statement.
- Fixed transaction leak on odb_transaction_commit() error.
Changes since V1:
- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling from odb_transaction_files_begin().
- odb_transaction_begin() now returns an error if there is already an inflight transaction pending instead of setting the out pointer to NULL.
- Updated odb_transaction_env() to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it from the_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.
Thanks,
-Justin
Justin Tobler (11):
object-file: rename files transaction prepare function
object-file: rename files transaction fsync function
object-file: embed transaction flush logic in commit function
object-file: drop check for inflight transactions
object-file: propagate files transaction errors odb/transaction: propagate begin errors odb/transaction: propagate commit errors odb/transaction: add transaction env interface odb/transaction: introduce ODB transaction flags builtin/receive-pack: drop redundant tmpdir env builtin/receive-pack: stage incoming objects via ODB transactions
builtin/add.c | 2 +- builtin/receive-pack.c | 69 ++++++++--------- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 7 +- object-file.c | 161 +++++++++++++++++++++++++-------------- object-file.h | 8 +- odb/source-files.c | 9 +-- odb/source-inmemory.c | 3 +- odb/source-loose.c | 3 +- odb/source.h | 9 ++- odb/transaction.c | 33 ++++++-- odb/transaction.h | 59 +++++++++++--- read-cache.c | 7 +- 14 files changed, 244 insertions(+), 130 deletions(-)
Range-diff against v3:
1: 9c14b219ad = 1: 9c14b219ad object-file: rename files transaction prepare function 2: 5703a9e93b = 2: 5703a9e93b object-file: rename files transaction fsync function 3: 76204847f2 ! 3: 70267741b0 object-file: embed transaction flush logic in commit function
@@ Commit message
subsequent commit, the transaction temporary directory is used to stage packfiles and not just loose objects anymore.
- Lift the helper function logic directly into
- odb_transaction_files_commit() to more accurately signal to readers
- the operation being performed.
+ Lift the helper function logic into odb_transaction_files_commit() to
+ more accurately signal to readers the operation being performed.
4: c97eb7763f = 4: 34cd3822c5 object-file: drop check for inflight transactions
5: 1f3a1f7714 = 5: 240aa3475f object-file: propagate files transaction errors
6: 09d13272d5 ! 6: 0d91310fac odb/transaction: propagate begin errors
@@ odb/transaction.c
- odb_source_begin_transaction(odb->sources, &odb->transaction);
+ ret = odb_source_begin_transaction(odb->sources, out);
-+ odb->transaction = *out;
++ if (!ret)
++ odb->transaction = *out;
- return odb->transaction;
+ return ret;
7: 12833d6773 = 7: 5e4680ed75 odb/transaction: propagate commit errors
8: f2586f2f34 ! 8: babcf6b156 odb/transaction: add transaction env interface
@@ object-file.c: static int odb_transaction_files_commit(struct odb_transaction *b
+{
+ struct odb_transaction_files *transaction =
+ container_of(base, struct odb_transaction_files, base);
++ int ret;
+
-+ odb_transaction_files_prepare(&transaction->base);
-+ strvec_pushv(env, tmp_objdir_env(transaction->objdir));
++ ret = odb_transaction_files_prepare(&transaction->base);
++ if (!ret)
++ strvec_pushv(env, tmp_objdir_env(transaction->objdir));
+
-+ return 0;
++ return ret;
+}
+
int odb_transaction_files_begin(struct odb_source *source, 9: 9d082b5e47 ! 9: 96f2a21eec odb/transaction: introduce ODB transaction flags
@@ odb/transaction.c
- ret = odb_source_begin_transaction(odb->sources, out);
+ ret = odb_source_begin_transaction(odb->sources, out, flags);
- odb->transaction = *out;
+ if (!ret)
+ odb->transaction = *out;
- return ret;
## odb/transaction.h ##
@@
10: e11d8a6676 = 10: 56718f1190 builtin/receive-pack: drop redundant tmpdir env 11: fee57c2817 = 11: 5197a19fbf builtin/receive-pack: stage incoming objects via ODB transactions
base-commit: ab776a62a78576513ee121424adb19597fbb7613
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 1/11 object-file: rename files transaction prepare function
The "files" ODB transaction backend lazily creates a temporary object directory when the first loose object is written to the transaction via prepare_loose_object_transaction(). In a subsequent commit, the temporary directory is used to also write packfiles to.
Rename the function to odb_transaction_files_prepare() accordingly.
--- object-file.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
object-file.c +3 -3
@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; }; -static void prepare_loose_object_transaction (struct odb_transaction *base) +static void odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);
@@ -761,7 +761,7 @@ int write_loose_object(struct odb_source_loose *loose,static struct strbuf filename =STRBUF_INIT ;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -prepare_loose_object_transaction (loose->base.odb->transaction); +odb_transaction_files_prepare (loose->base.odb->transaction);odb_loose_path (loose, &filename, oid);
@@ -825,7 +825,7 @@ int odb_source_loose_write_stream(struct odb_source_loose *loose,int hdrlen;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -prepare_loose_object_transaction (loose->base.odb->transaction); +odb_transaction_files_prepare (loose->base.odb->transaction);/* Since oid is not determined, save tmp file to odb path. */ strbuf_addf (&filename,"%s/" , loose->base.path);
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 2/11 object-file: rename files transaction fsync function
In reply to Justin ToblerWhen writing an object to a "files" ODB transaction, a full hardware flush is not initially performed during the fsync in fsync_loose_object_transaction() and instead delayed until the transaction is later committed.
To be more consistent with other "files" ODB transaction helpers, rename the function to odb_transaction_files_fsync() accordingly. The conditional in the helper is also slightly restructured to improve clarity to readers.
--- object-file.c | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-)
object-file.c +10 -6
@@ -518,12 +518,17 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)tmp_objdir_replace_primary_odb (transaction->objdir,0 ); } -static void fsync_loose_object_transaction (struct odb_transaction *base, -int fd,const char *filename) +static void odb_transaction_files_fsync (struct odb_transaction *base, +int fd,const char *filename) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base); +if (!transaction || !transaction->objdir) { +fsync_or_die (fd, filename); +return ; + } +/* * If we have an active ODB transaction, we issue a call that * cleans the filesystem page cache but avoids a hardware flush
@@ -531,8 +536,7 @@ static void fsync_loose_object_transaction(struct odb_transaction *base,* before renaming the objects to their final names as part of * flush_batch_fsync. */ -if (!transaction || !transaction->objdir || -git_fsync (fd,FSYNC_WRITEOUT_ONLY ) <0 ) { +if (git_fsync (fd,FSYNC_WRITEOUT_ONLY ) <0 ) {if (errno ==ENOSYS )warning (_ ("core.fsyncMethod = batch is unsupported on this platform" ));fsync_or_die (fd, filename);
@@ -553,7 +557,7 @@ static void flush_loose_object_transaction(struct odb_transaction_files *transac/* * Issue a full hardware flush against a temporary file to ensure * that all objects are durable before any renames occur. The code in -* fsync_loose_object_transaction has already issued a writeout +* odb_transaction_files_fsync has already issued a writeout * request, but it has not flushed any writeback cache in the storage * hardware or any filesystem logs. This fsync call acts as a barrier * to ensure that the data in each new object file is durable before
@@ -582,7 +586,7 @@ static void close_loose_object(struct odb_source_loose *loose, goto out;if (batch_fsync_enabled (FSYNC_COMPONENT_LOOSE_OBJECT )) -fsync_loose_object_transaction (loose->base.odb->transaction, fd, filename); +odb_transaction_files_fsync (loose->base.odb->transaction, fd, filename);else if (fsync_object_files >0 )fsync_or_die (fd, filename);else
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 3/11 object-file: embed transaction flush logic in commit function
In reply to Justin ToblerWhen a "files" transaction is committed, flush_loose_object_transaction() is invoked to handle performing a hardware flush along with migrating the temporary object directory into the primary and configuring the repository ODB source accordingly. The function name here is a bit misleading because the helper is doing a bit more than just "flushing" the transaction contents. Also, in a subsequent commit, the transaction temporary directory is used to stage packfiles and not just loose objects anymore.
Lift the helper function logic into odb_transaction_files_commit() to more accurately signal to readers the operation being performed.
--- object-file.c | 64 ++++++++++++++++++++++----------------------------- 1 file changed, 28 insertions(+), 36 deletions(-)
object-file.c +28 -36
@@ -543,41 +543,6 @@ static void odb_transaction_files_fsync(struct odb_transaction *base, } } -/* -* Cleanup after batch-mode fsync_object_files. -*/ -static void flush_loose_object_transaction (struct odb_transaction_files *transaction) -{ -struct strbuf temp_path =STRBUF_INIT ; -struct tempfile *temp; - -if (!transaction->objdir) -return ; - -/* -* Issue a full hardware flush against a temporary file to ensure -* that all objects are durable before any renames occur. The code in -* odb_transaction_files_fsync has already issued a writeout -* request, but it has not flushed any writeback cache in the storage -* hardware or any filesystem logs. This fsync call acts as a barrier -* to ensure that the data in each new object file is durable before -* the final name is visible. -*/ -strbuf_addf (&temp_path,"%s/bulk_fsync_XXXXXX" , -repo_get_object_directory (transaction->base.source->odb->repo)); - temp =xmks_tempfile (temp_path.buf); -fsync_or_die (get_tempfile_fd (temp),get_tempfile_path (temp)); -delete_tempfile (&temp); -strbuf_release (&temp_path); - -/* -* Make the object files visible in the primary ODB after their data is -* fully durable. -*/ -tmp_objdir_migrate (transaction->objdir); - transaction->objdir =NULL ; -} -/* Finalize a file on disk, and close it. */ static void close_loose_object (struct odb_source_loose *loose,int fd,const char *filename)
@@ -1679,7 +1644,34 @@ static void odb_transaction_files_commit(struct odb_transaction *base)struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base); -flush_loose_object_transaction (transaction); +if (transaction->objdir) { +struct strbuf temp_path =STRBUF_INIT ; +struct tempfile *temp; + +/* +* Issue a full hardware flush against a temporary file to ensure +* that all objects are durable before any renames occur. The code in +* odb_transaction_files_fsync has already issued a writeout +* request, but it has not flushed any writeback cache in the storage +* hardware or any filesystem logs. This fsync call acts as a barrier +* to ensure that the data in each new object file is durable before +* the final name is visible. +*/ +strbuf_addf (&temp_path,"%s/bulk_fsync_XXXXXX" , +repo_get_object_directory (transaction->base.source->odb->repo)); + temp =xmks_tempfile (temp_path.buf); +fsync_or_die (get_tempfile_fd (temp),get_tempfile_path (temp)); +delete_tempfile (&temp); +strbuf_release (&temp_path); + +/* +* Make the object files visible in the primary ODB after their data is +* fully durable. +*/ +tmp_objdir_migrate (transaction->objdir); + transaction->objdir =NULL ; + } +flush_packfile_transaction (transaction); }
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 4/11 object-file: drop check for inflight transactions
In reply to Justin ToblerODB transactions are started via odb_transaction_begin() and contain validation to avoid starting multiple transactions at the same time. The "files" backend also has the same logic, but is redundant due to the generic layer already handling it. Drop this validation from the "files" backend accordingly.
--- object-file.c | 4 ---- object-file.h | 3 +-- 2 files changed, 1 insertion(+), 6 deletions(-)
object-file.c +0 -4
@@ -1678,10 +1678,6 @@ static void odb_transaction_files_commit(struct odb_transaction *base)struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) {struct odb_transaction_files *transaction; -struct object_database *odb = source->odb; - -if (odb->transaction) -return NULL ; transaction =xcalloc (1 ,sizeof (*transaction)); transaction->base.source = source;
object-file.h +1 -2
@@ -194,8 +194,7 @@ struct odb_transaction;/* * Tell the object database to optimize for adding * multiple objects. odb_transaction_files_commit must be called -* to make new objects visible. If a transaction is already -* pending, NULL is returned. +* to make new objects visible. */ struct odb_transaction *odb_transaction_files_begin (struct odb_source *source);
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 5/11 object-file: propagate files transaction errors
In reply to Justin ToblerThe "files" transaction backend may encounter errors related to managing the temporary directory used to stage objects, but silently ignores these errors. Instead return errors encountered in the odb_transaction_files_{prepare,begin,commit}() interfaces to allow callers to handle them as needed.
--- object-file.c | 26 ++++++++++++++++++-------- object-file.h | 3 ++- odb/source-files.c | 6 +----- odb/transaction.h | 7 +++++-- 4 files changed, 26 insertions(+), 16 deletions(-)
object-file.c +18 -8
@@ -499,7 +499,7 @@ struct odb_transaction_files {struct transaction_packfile packfile; }; -static void odb_transaction_files_prepare (struct odb_transaction *base) +static int odb_transaction_files_prepare (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of_or_null(base, struct odb_transaction_files, base);
@@ -511,11 +511,15 @@ static void odb_transaction_files_prepare(struct odb_transaction *base)* added at the time they call odb_transaction_files_begin. */ if (!transaction || transaction->objdir) -return ; +return 0 ; transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); -if (transaction->objdir) -tmp_objdir_replace_primary_odb (transaction->objdir,0 ); +if (!transaction->objdir) +return error (_ ("unable to create temporary object directory" )); + +tmp_objdir_replace_primary_odb (transaction->objdir,0 ); + +return 0 ; } static void odb_transaction_files_fsync(struct odb_transaction *base,
@@ -1639,7 +1643,7 @@ int read_loose_object(struct repository *repo,return ret; } -static void odb_transaction_files_commit (struct odb_transaction *base) +static int odb_transaction_files_commit (struct odb_transaction *base) {struct odb_transaction_files *transaction = container_of(base, struct odb_transaction_files, base);
@@ -1668,14 +1672,19 @@ static void odb_transaction_files_commit(struct odb_transaction *base)* Make the object files visible in the primary ODB after their data is * fully durable. */ -tmp_objdir_migrate (transaction->objdir); +if (tmp_objdir_migrate (transaction->objdir)) +return error (_ ("unable to migrate temporary objects" )); + transaction->objdir =NULL ; }flush_packfile_transaction (transaction); + +return 0 ; } -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source) +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out) {struct odb_transaction_files *transaction;
@@ -1683,6 +1692,7 @@ struct odb_transaction *odb_transaction_files_begin(struct odb_source *source) transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + *out = &transaction->base; -return &transaction->base; +return 0 ; }
object-file.h +2 -1
@@ -196,6 +196,7 @@ struct odb_transaction;* multiple objects. odb_transaction_files_commit must be called * to make new objects visible. */ -struct odb_transaction *odb_transaction_files_begin (struct odb_source *source); +int odb_transaction_files_begin (struct odb_source *source, +struct odb_transaction **out);#endif /* OBJECT_FILE_H */
odb/source-files.c +1 -5
@@ -182,11 +182,7 @@ static int odb_source_files_write_object_stream(struct odb_source *source,static int odb_source_files_begin_transaction (struct odb_source *source,struct odb_transaction **out) { -struct odb_transaction *tx =odb_transaction_files_begin (source); -if (!tx) -return -1 ; - *out = tx; -return 0 ; +return odb_transaction_files_begin (source, out); } static int odb_source_files_read_alternates(struct odb_source *source,
odb/transaction.h +5 -2
@@ -16,8 +16,11 @@ struct odb_transaction {/* The ODB source the transaction is opened against. */ struct odb_source *source; -/* The ODB source specific callback invoked to commit a transaction. */ -void (*commit)(struct odb_transaction *transaction); +/* +* The ODB source specific callback invoked to commit a transaction. +* Returns 0 on success, a negative error code otherwise. +*/ +int (*commit)(struct odb_transaction *transaction);/* * This callback is expected to write the given object stream into
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 6/11 odb/transaction: propagate begin errors
In reply to Justin ToblerWhen odb_transaction_begin() is invoked, the function returns the transaction pointer directly. There is no way for the backend to signal that it failed to set up its state, such as when creating the temporary object directory backing the transaction.
In a subsequent commit, git-receive-pack(1) starts using ODB transactions and needs to be able to report such failures rather than silently ignore them. Refactor odb_transaction_begin() to return an int error code and write the resulting transaction into an out parameter. Also introduce odb_transaction_begin_or_die() as a convenience for callsites that do not need to handle errors explicitly.
Note that odb_transaction_begin() now returns an error when the ODB already has an inflight transaction pending. ODB transaction call sites that may encounter an inflight transaction are updated to explicitly handle this case.
--- builtin/add.c | 2 +- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 7 +++++-- object-file.c | 10 +++++++--- odb/transaction.c | 14 ++++++++++---- odb/transaction.h | 19 +++++++++++++++---- read-cache.c | 7 +++++-- 8 files changed, 45 insertions(+), 18 deletions(-)
builtin/add.c +1 -1
@@ -581,7 +581,7 @@ int cmd_add(int argc,string_list_clear (&only_match_skip_worktree,0 ); } - transaction =odb_transaction_begin (repo->objects); +odb_transaction_begin_or_die (repo->objects, &transaction); ps_matched =xcalloc (pathspec.nr,1 );if (add_renormalize)
builtin/unpack-objects.c +1 -1
@@ -598,7 +598,7 @@ static void unpack_all(void) progress =start_progress (the_repository,_ ("Unpacking objects" ), nr_objects);CALLOC_ARRAY (obj_list, nr_objects); - transaction =odb_transaction_begin (the_repository->objects); +odb_transaction_begin_or_die (the_repository->objects, &transaction);for (i =0 ; i < nr_objects; i++) {unpack_one (i);display_progress (progress, i +1 );
builtin/update-index.c +1 -1
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,* Allow the object layer to optimize adding multiple objects in * a batch. */ - transaction =odb_transaction_begin (the_repository->objects); +odb_transaction_begin_or_die (the_repository->objects, &transaction);while (ctx.argc) {if (parseopt_state !=PARSE_OPT_DONE ) parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +5 -2
@@ -474,6 +474,7 @@ static int update_one(struct cache_tree *it,int cache_tree_update (struct index_state *istate,int flags) { +int inflight = !!the_repository->objects->transaction;struct odb_transaction *transaction;int skip, i;
@@ -490,10 +491,12 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo); - transaction =odb_transaction_begin (the_repository->objects); +if (!inflight) +odb_transaction_begin_or_die (the_repository->objects, &transaction); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction);trace2_region_leave ("cache_tree" ,"update" , istate->repo);trace_performance_leave ("cache_tree_update" );if (i <0 )
object-file.c +7 -3
@@ -1354,13 +1354,17 @@ int index_fd(struct index_state *istate, struct object_id *oid,if (flags &INDEX_WRITE_OBJECT ) {struct object_database *odb = the_repository->objects; -struct odb_transaction *transaction =odb_transaction_begin (odb); +struct odb_transaction *transaction = odb->transaction; +int inflight = !!transaction; - ret =odb_transaction_write_object_stream (odb->transaction, +if (!inflight) +odb_transaction_begin_or_die (odb, &transaction); + ret =odb_transaction_write_object_stream (transaction, &stream,xsize_t (st->st_size), oid); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction); }else { ret = hash_blob_stream(&stream, the_repository->hash_algo, oid,
odb/transaction.c +10 -4
@@ -1,15 +1,21 @@#include "git-compat-util.h" +#include "gettext.h" #include "odb/source.h" #include "odb/transaction.h" -struct odb_transaction *odb_transaction_begin (struct object_database *odb) +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out) { +int ret; +if (odb->transaction) -return NULL ; +return error (_ ("object database transaction already pending" )); -odb_source_begin_transaction (odb->sources, &odb->transaction); + ret =odb_source_begin_transaction (odb->sources, out); +if (!ret) + odb->transaction = *out; -return odb->transaction; +return ret; }void odb_transaction_commit (struct odb_transaction *transaction)
odb/transaction.h +15 -4
@@ -1,6 +1,7 @@#ifndef ODB_TRANSACTION_H #define ODB_TRANSACTION_H +#include "gettext.h" #include "odb.h" #include "odb/source.h"
@@ -36,11 +37,21 @@ struct odb_transaction { };/* -* Starts an ODB transaction. Subsequent objects are written to the transaction -* and not committed until odb_transaction_commit() is invoked on the -* transaction. If the ODB already has a pending transaction, NULL is returned. +* Starts an ODB transaction and returns it via `out`. Subsequent objects are +* written to the transaction and not committed until odb_transaction_commit() +* is invoked on the transaction. Returns 0 on success and a negative value on +* error. Note that it is considered an error to start a new transaction if the +* ODB already has an inflight transaction pending. */ -struct odb_transaction *odb_transaction_begin (struct object_database *odb); +int odb_transaction_begin (struct object_database *odb, +struct odb_transaction **out); + +static inline void odb_transaction_begin_or_die (struct object_database *odb, +struct odb_transaction **out) +{ +if (odb_transaction_begin (odb, out)) +die (_ ("failed to start ODB transaction" )); +}/* * Commits an ODB transaction making the written objects visible. If the
read-cache.c +5 -2
@@ -4012,6 +4012,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix, const struct pathspec *pathspec, char *ps_matched, int include_sparse, int flags, int ignored_too ) { + int inflight = !!repo->objects->transaction;struct odb_transaction *transaction;struct update_callback_data data;struct rev_info rev;
@@ -4042,9 +4043,11 @@ int add_files_to_cache(struct repository *repo, const char *prefix,* This function is invoked from commands other than 'add', which * may not have their own transaction active. */ - transaction =odb_transaction_begin (repo->objects); +if (!inflight) +odb_transaction_begin_or_die (repo->objects, &transaction);run_diff_files (&rev,DIFF_RACY_IS_MODIFIED ); -odb_transaction_commit (transaction); +if (!inflight) +odb_transaction_commit (transaction);release_revisions (&rev);return !!data.add_errors;
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 7/11 odb/transaction: propagate commit errors
In reply to Justin ToblerWhen odb_transaction_commit() is invoked, the return value of the backend commit callback is silently discarded. A backend has no way to signal that committing failed, such as when the "files" backend cannot migrate its temporary object directory into the permanent ODB.
In a subsequent commit, git-receive-pack(1) starts using ODB transaction to stage objects and consequently cares about such failures so it can handle the error appropriately. Change the commit callback signature to return an int error code and have odb_transaction_commit() forward it accordingly.
--- odb/transaction.c | 10 +++++++--- odb/transaction.h | 7 ++++--- 2 files changed, 11 insertions(+), 6 deletions(-)
odb/transaction.c +7 -3
@@ -18,19 +18,23 @@ int odb_transaction_begin(struct object_database *odb,return ret; } -void odb_transaction_commit (struct odb_transaction *transaction) +int odb_transaction_commit (struct odb_transaction *transaction) { +int ret; +if (!transaction) -return ; +return 0 ;/* * Ensure the transaction ending matches the pending transaction. */ ASSERT (transaction == transaction->source->odb->transaction); - transaction->commit (transaction); + ret = transaction->commit (transaction); transaction->source->odb->transaction =NULL ;free (transaction); + +return ret; } int odb_transaction_write_object_stream(struct odb_transaction *transaction,
odb/transaction.h +4 -3
@@ -54,10 +54,11 @@ static inline void odb_transaction_begin_or_die(struct object_database *odb, }/* -* Commits an ODB transaction making the written objects visible. If the -* specified transaction is NULL, the function is a no-op. +* Commits an ODB transaction making the written objects visible. Returns 0 on +* success, a negative error code otherwise. Note that, if the specified +* transaction is NULL, the function is a no-op and no error is returned. */ -void odb_transaction_commit (struct odb_transaction *transaction); +int odb_transaction_commit (struct odb_transaction *transaction);/* * Writes the object in the provided stream into the transaction. The resulting
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 8/11 odb/transaction: add transaction env interface
In reply to Justin ToblerThe ODB transaction backend is responsible for creating/managing its own staging area for writing objects. Other child processes spawned by Git may need access to uncommitted objects or write new objects in the staging area though.
Introduce odb_transaction_env() which is expected to provide the set of environment variables needed by a child process to access the transaction's staging area.
--- object-file.c | 16 ++++++++++++++++ odb/transaction.c | 8 ++++++++ odb/transaction.h | 17 +++++++++++++++++ 3 files changed, 41 insertions(+)
object-file.c +16 -0
@@ -27,6 +27,7 @@#include "path.h" #include "read-cache-ll.h" #include "setup.h" +#include "strvec.h" #include "tempfile.h" #include "tmp-objdir.h"
@@ -1687,6 +1688,20 @@ static int odb_transaction_files_commit(struct odb_transaction *base)return 0 ; } +static int odb_transaction_files_env (struct odb_transaction *base, +struct strvec *env) +{ +struct odb_transaction_files *transaction = + container_of(base, struct odb_transaction_files, base); +int ret; + + ret =odb_transaction_files_prepare (&transaction->base); +if (!ret) +strvec_pushv (env,tmp_objdir_env (transaction->objdir)); + +return ret; +} +int odb_transaction_files_begin (struct odb_source *source,struct odb_transaction **out) {
@@ -1696,6 +1711,7 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.source = source; transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; + transaction->base.env = odb_transaction_files_env; *out = &transaction->base;return 0 ;
odb/transaction.c +8 -0
@@ -43,3 +43,11 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction, {return transaction->write_object_stream (transaction, stream, len, oid); } + +int odb_transaction_env (struct odb_transaction *transaction,struct strvec *env) +{ +if (!transaction) +return 0 ; + +return transaction->env (transaction, env); +}
odb/transaction.h +17 -0
@@ -34,6 +34,14 @@ struct odb_transaction {int (*write_object_stream)(struct odb_transaction *transaction,struct odb_write_stream *stream,size_t len,struct object_id *oid); + +/* +* This callback is expected to populate the provided strvec with the +* environment variables that a child process should inherit so that its +* object writes participate in the transaction. Returns 0 on success, a +* negative error code otherwise. +*/ +int (*env)(struct odb_transaction *transaction,struct strvec *env); };/*
@@ -69,4 +77,13 @@ int odb_transaction_write_object_stream(struct odb_transaction *transaction,struct odb_write_stream *stream, size_t len, struct object_id *oid); +/* +* Populates the provided strvec with the environment variables that a child +* process should inherit so that its object writes participate in the +* transaction, suitable for using via child_process.env. Returns 0 on success, +* a negative error code otherwise. Note that, if the specified transaction is +* NULL, the function is a no-op and no error is returned. +*/ +int odb_transaction_env (struct odb_transaction *transaction,struct strvec *env); +#endif
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 9/11 odb/transaction: introduce ODB transaction flags
In reply to Justin ToblerThe temporary directory used by git-receive-pack(1) to write objects is managed slightly differently than how it is done via ODB transactions:
- The temporary directory is eagerly created upfront, instead of waiting for the first object write.
- The prefix name of the temporary directory is "incoming" instead of
"bulk-fsync".
In a subsequent commit, git-receive-pack(1) will use ODB transactions instead of tmp_objdir directly. To provide a means to configure the same transaction behavior, introduce enum odb_transaction_flags and the ODB_TRANSACTION_RECEIVE flag intended as a signal for ODB transactions using the "files" backend to be set up for git-receive-pack(1). Transaction call sites are updated accordingly to provide the required flag parameter.
--- builtin/add.c | 2 +- builtin/unpack-objects.c | 2 +- builtin/update-index.c | 2 +- cache-tree.c | 2 +- object-file.c | 29 ++++++++++++++++++++++++++--- object-file.h | 4 +++- odb/source-files.c | 5 +++-- odb/source-inmemory.c | 3 ++- odb/source-loose.c | 3 ++- odb/source.h | 9 ++++++--- odb/transaction.c | 5 +++-- odb/transaction.h | 15 +++++++++++---- read-cache.c | 2 +- 13 files changed, 61 insertions(+), 22 deletions(-)
builtin/add.c +1 -1
@@ -581,7 +581,7 @@ int cmd_add(int argc,string_list_clear (&only_match_skip_worktree,0 ); } -odb_transaction_begin_or_die (repo->objects, &transaction); +odb_transaction_begin_or_die (repo->objects, &transaction,0 ); ps_matched =xcalloc (pathspec.nr,1 );if (add_renormalize)
builtin/unpack-objects.c +1 -1
@@ -598,7 +598,7 @@ static void unpack_all(void) progress =start_progress (the_repository,_ ("Unpacking objects" ), nr_objects);CALLOC_ARRAY (obj_list, nr_objects); -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 );for (i =0 ; i < nr_objects; i++) {unpack_one (i);display_progress (progress, i +1 );
builtin/update-index.c +1 -1
@@ -1124,7 +1124,7 @@ int cmd_update_index(int argc,* Allow the object layer to optimize adding multiple objects in * a batch. */ -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 );while (ctx.argc) {if (parseopt_state !=PARSE_OPT_DONE ) parseopt_state = parse_options_step(&ctx, options,
cache-tree.c +1 -1
@@ -492,7 +492,7 @@ int cache_tree_update(struct index_state *istate, int flags)trace_performance_enter ();trace2_region_enter ("cache_tree" ,"update" , istate->repo);if (!inflight) -odb_transaction_begin_or_die (the_repository->objects, &transaction); +odb_transaction_begin_or_die (the_repository->objects, &transaction,0 ); i =update_one (istate->cache_tree, istate->cache, istate->cache_nr,"" ,0 , &skip, flags);if (!inflight)
object-file.c +26 -3
@@ -498,6 +498,7 @@ struct odb_transaction_files {struct tmp_objdir *objdir;struct transaction_packfile packfile; +const char *prefix; };static int odb_transaction_files_prepare (struct odb_transaction *base)
@@ -514,7 +515,7 @@ static int odb_transaction_files_prepare(struct odb_transaction *base)if (!transaction || transaction->objdir)return 0 ; - transaction->objdir =tmp_objdir_create (base->source->odb->repo,"bulk-fsync" ); + transaction->objdir =tmp_objdir_create (base->source->odb->repo, transaction->prefix);if (!transaction->objdir)return error (_ ("unable to create temporary object directory" ));
@@ -1359,7 +1360,7 @@ int index_fd(struct index_state *istate, struct object_id *oid,int inflight = !!transaction;if (!inflight) -odb_transaction_begin_or_die (odb, &transaction); +odb_transaction_begin_or_die (odb, &transaction,0 ); ret = odb_transaction_write_object_stream(transaction, &stream, xsize_t(st->st_size),
@@ -1703,7 +1704,8 @@ static int odb_transaction_files_env(struct odb_transaction *base, }int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {struct odb_transaction_files *transaction;
@@ -1712,6 +1714,27 @@ int odb_transaction_files_begin(struct odb_source *source, transaction->base.commit = odb_transaction_files_commit; transaction->base.write_object_stream = odb_transaction_files_write_object_stream; transaction->base.env = odb_transaction_files_env; + + transaction->prefix ="bulk-fsync" ; +if (flags &ODB_TRANSACTION_RECEIVE ) { +/* +* ODB transactions for git-receive-pack(1) eagerly create a +* temporary directory and use a different temporary directory +* prefix. +* +* NEEDSWORK: This transaction flag is only used by the "files" +* backend to special case temporary directory set up and +* handling. Ideally transaction users should not have to care +* though. To avoid this, we could eagerly create the temporary +* directory and use the same prefix name for all transactions. +*/ + transaction->prefix ="incoming" ; +if (odb_transaction_files_prepare (&transaction->base)) { +free (transaction); +return -1 ; + } + } + *out = &transaction->base;return 0 ;
object-file.h +3 -1
@@ -5,6 +5,7 @@#include "object.h" #include "odb.h" #include "odb/source-loose.h" +#include "odb/transaction.h" /* The maximum size for an object header. */ #define MAX_HEADER_LEN 32
@@ -197,6 +198,7 @@ struct odb_transaction;* to make new objects visible. */ int odb_transaction_files_begin (struct odb_source *source, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);#endif /* OBJECT_FILE_H */
odb/source-files.c +3 -2
@@ -180,9 +180,10 @@ static int odb_source_files_write_object_stream(struct odb_source *source, }static int odb_source_files_begin_transaction (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -return odb_transaction_files_begin (source, out); +return odb_transaction_files_begin (source, out, flags); } static int odb_source_files_read_alternates(struct odb_source *source,
odb/source-inmemory.c +2 -1
@@ -304,7 +304,8 @@ static int odb_source_inmemory_freshen_object(struct odb_source *source, } static int odb_source_inmemory_begin_transaction(struct odb_source *source UNUSED, - struct odb_transaction **out UNUSED) + struct odb_transaction **out UNUSED, + enum odb_transaction_flags flags UNUSED) {return error ("in-memory source does not support transactions" ); }
odb/source-loose.c +2 -1
@@ -638,7 +638,8 @@ static int odb_source_loose_write_object_stream(struct odb_source *source, } static int odb_source_loose_begin_transaction(struct odb_source *source UNUSED, - struct odb_transaction **out UNUSED) + struct odb_transaction **out UNUSED, + enum odb_transaction_flags flags UNUSED) {/* TODO: this is a known omission that we'll want to address eventually. */ return error ("loose source does not support transactions" );
odb/source.h +6 -3
@@ -3,6 +3,7 @@#include "object.h" #include "odb.h" +#include "odb/transaction.h" enum odb_source_type {/*
@@ -228,7 +229,8 @@ struct odb_source {* negative error code otherwise. */ int (*begin_transaction)(struct odb_source *source, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);/* * This callback is expected to read the list of alternate object
@@ -467,9 +469,10 @@ static inline int odb_source_write_alternate(struct odb_source *source,* Returns 0 on success, a negative error code otherwise. */ static inline int odb_source_begin_transaction (struct odb_source *source, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -return source->begin_transaction (source, out); +return source->begin_transaction (source, out, flags); }#endif
odb/transaction.c +3 -2
@@ -4,14 +4,15 @@#include "odb/transaction.h" int odb_transaction_begin (struct object_database *odb, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) {int ret;if (odb->transaction)return error (_ ("object database transaction already pending" )); - ret =odb_source_begin_transaction (odb->sources, out); + ret =odb_source_begin_transaction (odb->sources, out, flags);if (!ret) odb->transaction = *out;
odb/transaction.h +11 -4
@@ -3,7 +3,6 @@#include "gettext.h" #include "odb.h" -#include "odb/source.h" /* * A transaction may be started for an object database prior to writing new
@@ -44,6 +43,12 @@ struct odb_transaction {int (*env)(struct odb_transaction *transaction,struct strvec *env); }; +/* Flags used to configure an ODB transaction. */ +enum odb_transaction_flags { +/* Configures the transaction for use with git-receive-pack(1). */ +ODB_TRANSACTION_RECEIVE = (1 <<0 ), +}; +/* * Starts an ODB transaction and returns it via `out`. Subsequent objects are * written to the transaction and not committed until odb_transaction_commit()
@@ -52,12 +57,14 @@ struct odb_transaction {* ODB already has an inflight transaction pending. */ int odb_transaction_begin (struct object_database *odb, -struct odb_transaction **out); +struct odb_transaction **out, +enum odb_transaction_flags flags);static inline void odb_transaction_begin_or_die (struct object_database *odb, -struct odb_transaction **out) +struct odb_transaction **out, +enum odb_transaction_flags flags) { -if (odb_transaction_begin (odb, out)) +if (odb_transaction_begin (odb, out, flags))die (_ ("failed to start ODB transaction" )); }
read-cache.c +1 -1
@@ -4044,7 +4044,7 @@ int add_files_to_cache(struct repository *repo, const char *prefix,* may not have their own transaction active. */ if (!inflight) -odb_transaction_begin_or_die (repo->objects, &transaction); +odb_transaction_begin_or_die (repo->objects, &transaction,0 );run_diff_files (&rev,DIFF_RACY_IS_MODIFIED );if (!inflight)odb_transaction_commit (transaction);
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 10/11 builtin/receive-pack: drop redundant tmpdir env
In reply to Justin ToblerWhen performing the connectivity checks for a shallow ref in update_shallow_ref(), the child process environment variables are populated via tmp_objdir_env(). This is unnecessary though as update_shallow_ref() is only reached after tmp_objdir_migrate() has been performed which means there is no longer a temporary directory that needs to be shared with child processes.
Drop the call to tmp_objdir_env() accordingly.
--- builtin/receive-pack.c | 1 - 1 file changed, 1 deletion(-)
builtin/receive-pack.c +0 -1
@@ -1363,7 +1363,6 @@ static int update_shallow_ref(struct command *cmd, struct shallow_info *si) !delayed_reachability_test(si, i))oid_array_append (&extra, &si->shallow->oid[i]); - opt.env =tmp_objdir_env (tmp_objdir);setup_alternate_shallow (&shallow_lock, &opt.shallow_file, &extra);if (check_connected (command_singleton_iterator, cmd, &opt)) {rollback_shallow_file (the_repository, &shallow_lock);
Signature · 1 line
2.55.0.122.gf85a7e6620
PATCH v4 11/11 builtin/receive-pack: stage incoming objects via ODB transactions
In reply to Justin ToblerObjects received by git-receive-pack(1) are quarantined in a temporary "incoming" directory and migrated into the object database prior to the reference updates. The quarantine is currently managed through tmp_objdir directly. In a pluggable ODB future, how exactly an object gets written to a transaction may vary for a given ODB source. Refactor git-receive-pack(1) to use the ODB transaction interfaces to manage the object staging area in a more agnostic manner accordingly.
Note that the ODB transaction is now responsible for managing the primary and alternate ODBs for the repository. One small change as a result is that the temporary directory is now applied as the primary ODB in the main process instead of an alternate. This does not change anything for git-receive-pack(1) though because it only needs access to the newly written objects and doesn't care how exactly it is set up.
--- builtin/receive-pack.c | 68 ++++++++++++++++++++++-------------------- 1 file changed, 35 insertions(+), 33 deletions(-)
builtin/receive-pack.c +35 -33
@@ -37,7 +37,6 @@#include "sigchain.h" #include "string-list.h" #include "strvec.h" -#include "tmp-objdir.h" #include "trace.h" #include "trace2.h" #include "version.h"
@@ -112,8 +111,6 @@ static enum { } use_keepalive;static int keepalive_in_sec =5 ; -static struct tmp_objdir *tmp_objdir; - static struct proc_receive_ref { unsigned int want_add:1, want_delete:1,
@@ -926,6 +923,7 @@ static void receive_hook_feed_state_free(void *data)static int run_receive_hook (struct command *commands,const char *hook_name,int skip_broken, +struct odb_transaction *transaction,const struct string_list *push_options) {struct run_hooks_opt opt =RUN_HOOKS_OPT_INIT ;
@@ -959,8 +957,8 @@ static int run_receive_hook(struct command *commands,strvec_push (&opt.env,"GIT_PUSH_OPTION_COUNT" ); } -if (tmp_objdir) -strvec_pushv (&opt.env,tmp_objdir_env (tmp_objdir)); +if (transaction) +odb_transaction_env (transaction, &opt.env);prepare_push_cert_sha1 (&opt);
@@ -1789,24 +1787,30 @@ static const struct object_id *command_singleton_iterator(void *cb_data) }static void set_connectivity_errors (struct command *commands, -struct shallow_info *si) +struct shallow_info *si, +struct odb_transaction *transaction) {struct command *cmd;for (cmd = commands; cmd; cmd = cmd->next) {struct command *singleton = cmd;struct check_connected_options opt =CHECK_CONNECTED_INIT ; +struct strvec env =STRVEC_INIT ;if (shallow_update && si->shallow_ref[cmd->index])/* to be checked in update_shallow_ref() */ continue ; - opt.env =tmp_objdir_env (tmp_objdir); +odb_transaction_env (transaction, &env); + opt.env = env.v; +if (!check_connected (command_singleton_iterator, &singleton, &opt))continue ; cmd->error_string ="missing necessary objects" ; + +strvec_clear (&env); } }
@@ -2027,6 +2031,7 @@ static void execute_commands_atomic(struct command *commands,static void execute_commands (struct command *commands,const char *unpacker_error,struct shallow_info *si, +struct odb_transaction *transaction,const struct string_list *push_options) {struct check_connected_options opt =CHECK_CONNECTED_INIT ;
@@ -2043,6 +2048,8 @@ static void execute_commands(struct command *commands, }if (!skip_connectivity_check) { +struct strvec env =STRVEC_INIT ; +if (use_sideband) {memset (&muxer,0 ,sizeof (muxer)); muxer.proc = copy_to_sideband;
@@ -2056,14 +2063,17 @@ static void execute_commands(struct command *commands, data.si = si; opt.err_fd = err_fd; opt.progress = err_fd && !quiet; - opt.env =tmp_objdir_env (tmp_objdir); +odb_transaction_env (transaction, &env); + opt.env = env.v; opt.exclude_hidden_refs_section ="receive" ;if (check_connected (iterate_receive_command_list, &data, &opt)) -set_connectivity_errors (commands, si); +set_connectivity_errors (commands, si, transaction);if (use_sideband)finish_async (&muxer); + +strvec_clear (&env); }reject_updates_to_hidden (commands);
@@ -2084,7 +2094,7 @@ static void execute_commands(struct command *commands, } } -if (run_receive_hook (commands,"pre-receive" ,0 , push_options)) { +if (run_receive_hook (commands,"pre-receive" ,0 , transaction, push_options)) {for (cmd = commands; cmd; cmd = cmd->next) {if (!cmd->error_string) cmd->error_string ="pre-receive hook declined" ;
@@ -2105,14 +2115,13 @@ static void execute_commands(struct command *commands,* Now we'll start writing out refs, which means the objects need * to be in their final positions so that other processes can see them. */ -if (tmp_objdir_migrate (tmp_objdir) <0 ) { +if (odb_transaction_commit (transaction)) {for (cmd = commands; cmd; cmd = cmd->next) {if (!cmd->error_string) cmd->error_string ="unable to migrate objects to permanent storage" ; }return ; } - tmp_objdir =NULL ;check_aliased_updates (commands);
@@ -2325,7 +2334,8 @@ static void push_header_arg(struct strvec *args, struct pack_header *hdr) ntohl(hdr->hdr_version), ntohl(hdr->hdr_entries)); } -static const char *unpack (int err_fd,struct shallow_info *si) +static const char *unpack (int err_fd,struct shallow_info *si, +struct odb_transaction *transaction) {struct pack_header hdr;const char *hdr_err;
@@ -2350,20 +2360,7 @@ static const char *unpack(int err_fd, struct shallow_info *si)strvec_push (&child.args, alt_shallow_file); } - tmp_objdir =tmp_objdir_create (the_repository,"incoming" ); -if (!tmp_objdir) { -if (err_fd >0 ) -close (err_fd); -return "unable to create temporary object directory" ; - } -strvec_pushv (&child.env,tmp_objdir_env (tmp_objdir)); - -/* -* Normally we just pass the tmp_objdir environment to the child -* processes that do the heavy lifting, but we may need to see these -* objects ourselves to set up shallow information. -*/ -tmp_objdir_add_as_alternate (tmp_objdir); +odb_transaction_env (transaction, &child.env);if (ntohl (hdr.hdr_entries) < unpack_limit) {strvec_push (&child.args,"unpack-objects" );
@@ -2430,13 +2427,14 @@ static const char *unpack(int err_fd, struct shallow_info *si)return NULL ; } -static const char *unpack_with_sideband (struct shallow_info *si) +static const char *unpack_with_sideband (struct shallow_info *si, +struct odb_transaction *transaction) {struct async muxer;const char *ret;if (!use_sideband) -return unpack (0 , si); +return unpack (0 , si, transaction); use_keepalive =KEEPALIVE_AFTER_NUL ;memset (&muxer,0 ,sizeof (muxer));
@@ -2445,7 +2443,7 @@ static const char *unpack_with_sideband(struct shallow_info *si)if (start_async (&muxer))return NULL ; - ret =unpack (muxer.in, si); + ret =unpack (muxer.in, si, transaction);finish_async (&muxer);return ret;
@@ -2622,6 +2620,7 @@ int cmd_receive_pack(int argc,struct oid_array ref =OID_ARRAY_INIT ;struct shallow_info si;struct packet_reader reader; +struct odb_transaction *transaction =NULL ;struct option options[] = {OPT__QUIET (&quiet,N_ ("quiet" )),
@@ -2706,11 +2705,14 @@ int cmd_receive_pack(int argc,if (!si.nr_ours && !si.nr_theirs) shallow_update =0 ;if (!delete_only (commands)) { - unpack_status =unpack_with_sideband (&si); +if (odb_transaction_begin (the_repository->objects, &transaction,ODB_TRANSACTION_RECEIVE )) + unpack_status ="unable to start object transaction" ; +else + unpack_status =unpack_with_sideband (&si, transaction);update_shallow_info (commands, &si, &ref); } use_keepalive =KEEPALIVE_ALWAYS ; -execute_commands (commands, unpack_status, &si, +execute_commands (commands, unpack_status, &si, transaction, &push_options);delete_tempfile (&pack_lockfile);sigchain_push (SIGPIPE ,SIG_IGN );
@@ -2719,7 +2721,7 @@ int cmd_receive_pack(int argc,else if (report_status)report (commands, unpack_status);sigchain_pop (SIGPIPE ); -run_receive_hook (commands,"post-receive" ,1 , +run_receive_hook (commands,"post-receive" ,1 ,NULL , &push_options);run_update_post_hook (commands);free_commands (commands);
Signature · 1 line
2.55.0.122.gf85a7e6620
Quoted 6 lines · Justin Tobler
Changes since V3:
- Removed ugly line break in commit message to prevent eye strain.
-odb_transaction_begin()now only sets the repository transaction on success.
-odb_transaction_env()now bubbles up error when failing to create the temporary directory.
Thanks, all the changes here look good to me and I'm happy with the state of this patch series.
Patrick