Skip to content

receive-pack: use ODB transactions to stage object writes

Justin Tobler git 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest11 patches 1 replyIn review
  2. v311 patches 9 repliesSuperseded
  3. v211 patches 16 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v16 patches21 repliesSuperseded

Changes in v4 from the cover letter

- Removed ugly line break in commit message to prevent eye strain.
- odb_transaction_begin() now only sets the repository transaction on success.
- odb_transaction_env() now bubbles up error when failing to create the temporary directory.

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/11object-file: rename files transaction prepare functionobject-file.c+3 -30 replies
2/11object-file: rename files transaction fsync functionobject-file.c+10 -60 replies
3/11object-file: embed transaction flush logic in commit functionobject-file.c+28 -360 replies
4/11object-file: drop check for inflight transactionsobject-file.c object-file.h+1 -60 replies
5/11object-file: propagate files transaction errorsobject-file.c object-file.h odb/source-files.c +1 more+26 -160 replies
6/11odb/transaction: propagate begin errorsbuiltin/add.c builtin/unpack-objects.c builtin/update-index.c +5 more+45 -180 replies
7/11odb/transaction: propagate commit errorsodb/transaction.c odb/transaction.h+11 -60 replies
8/11odb/transaction: add transaction env interfaceobject-file.c odb/transaction.c odb/transaction.h+41 -00 replies
9/11odb/transaction: introduce ODB transaction flagsbuiltin/add.c builtin/unpack-objects.c builtin/update-index.c +10 more+61 -220 replies
10/11builtin/receive-pack: drop redundant tmpdir envbuiltin/receive-pack.c+0 -10 replies
11/11builtin/receive-pack: stage incoming objects via ODB transactionsbuiltin/receive-pack.c+35 -330 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v4 yet.

Read the v4 thread

Cover letter [PATCH v4 00/11] receive-pack: use ODB transactions to stage object writes

Greetings,

This patch series replaces direct usage of the tmp_objdir interfaces in git-receive-pack(1) to instead use the odb_transaction interfaces to create/manage a staging area to write objects to. The purpose of this change is to get git-receive-pack(1) one step closer to being ODB backend agnostic. For now, the object writes themselves are still "files" backend specific due to being handled by the git-index-pack(1) and git-unpack-objects(1) child processes. This will be tackled in a separate series though.

Changes since V3:
- Removed ugly line break in commit message to prevent eye strain.
- odb_transaction_begin() now only sets the repository transaction on success.
- odb_transaction_env() now bubbles up error when failing to create the temporary directory.

Changes since V2:
- Clarified commit log reasoning for embedding flush_loose_object_transaction() logic in commit function.
- Started printed some error messages on transaction errors.
- Removed include statement.
- Fixed transaction leak on odb_transaction_commit() error.

Changes since V1:
- Adapted other "file" ODB transaction helpers to be more consistent with current naming scheme.
- Removed redundant NULL transaction handling from odb_transaction_files_begin().
- odb_transaction_begin() now returns an error if there is already an inflight transaction pending instead of setting the out pointer to NULL.
- Updated odb_transaction_env() to return an error code and append environment variables to a strvec provided as an argument.
- Removed redundant setting of tmpdir environment variables for child processes after tmpdir has been migrated.
- Split changes adding ODB transaction flags into a separate commit.
- Consistently wire the ODB transaction throughout git-receive-pack code instead of reading it from the_repository.
- Updated user facing error message.
- Updated some comments to better document functions/flags.
- Clarified some commit messages.
- Fixed typos.

Thanks,
-Justin

Justin Tobler (11):
object-file: rename files transaction prepare function
object-file: rename files transaction fsync function
object-file: embed transaction flush logic in commit function
object-file: drop check for inflight transactions
object-file: propagate files transaction errors odb/transaction: propagate begin errors odb/transaction: propagate commit errors odb/transaction: add transaction env interface odb/transaction: introduce ODB transaction flags builtin/receive-pack: drop redundant tmpdir env builtin/receive-pack: stage incoming objects via ODB transactions

 builtin/add.c            |   2 +-
 builtin/receive-pack.c   |  69 ++++++++---------
 builtin/unpack-objects.c |   2 +-
 builtin/update-index.c   |   2 +-
 cache-tree.c             |   7 +-
 object-file.c            | 161 +++++++++++++++++++++++++--------------
 object-file.h            |   8 +-
 odb/source-files.c       |   9 +--
 odb/source-inmemory.c    |   3 +-
 odb/source-loose.c       |   3 +-
 odb/source.h             |   9 ++-
 odb/transaction.c        |  33 ++++++--
 odb/transaction.h        |  59 +++++++++++---
 read-cache.c             |   7 +-
 14 files changed, 244 insertions(+), 130 deletions(-)

Range-diff against v3:

 1:  9c14b219ad =  1:  9c14b219ad object-file: rename files transaction prepare function
 2:  5703a9e93b =  2:  5703a9e93b object-file: rename files transaction fsync function
 3:  76204847f2 !  3:  70267741b0 object-file: embed transaction flush logic in commit function

@@ Commit message
subsequent commit, the transaction temporary directory is used to stage packfiles and not just loose objects anymore.

- Lift the helper function logic directly into
- odb_transaction_files_commit() to more accurately signal to readers
- the operation being performed.
+ Lift the helper function logic into odb_transaction_files_commit() to
+ more accurately signal to readers the operation being performed.

 4:  c97eb7763f =  4:  34cd3822c5 object-file: drop check for inflight transactions
 5:  1f3a1f7714 =  5:  240aa3475f object-file: propagate files transaction errors
 6:  09d13272d5 !  6:  0d91310fac odb/transaction: propagate begin errors
    @@ odb/transaction.c
      
     -	odb_source_begin_transaction(odb->sources, &odb->transaction);
     +	ret = odb_source_begin_transaction(odb->sources, out);
    -+	odb->transaction = *out;
    ++	if (!ret)
    ++		odb->transaction = *out;
      
     -	return odb->transaction;
     +	return ret;
 7:  12833d6773 =  7:  5e4680ed75 odb/transaction: propagate commit errors
 8:  f2586f2f34 !  8:  babcf6b156 odb/transaction: add transaction env interface
    @@ object-file.c: static int odb_transaction_files_commit(struct odb_transaction *b
     +{
     +	struct odb_transaction_files *transaction =
     +		container_of(base, struct odb_transaction_files, base);
    ++	int ret;
     +
    -+	odb_transaction_files_prepare(&transaction->base);
    -+	strvec_pushv(env, tmp_objdir_env(transaction->objdir));
    ++	ret = odb_transaction_files_prepare(&transaction->base);
    ++	if (!ret)
    ++		strvec_pushv(env, tmp_objdir_env(transaction->objdir));
     +
    -+	return 0;
    ++	return ret;
     +}
     +
      int odb_transaction_files_begin(struct odb_source *source,

9: 9d082b5e47 ! 9: 96f2a21eec odb/transaction: introduce ODB transaction flags
@@ odb/transaction.c

- ret = odb_source_begin_transaction(odb->sources, out);
+ ret = odb_source_begin_transaction(odb->sources, out, flags);
- odb->transaction = *out;
+ if (!ret)
+ odb->transaction = *out;

- return ret;

## odb/transaction.h ##
@@

10:  e11d8a6676 = 10:  56718f1190 builtin/receive-pack: drop redundant tmpdir env
11:  fee57c2817 = 11:  5197a19fbf builtin/receive-pack: stage incoming objects via ODB transactions

base-commit: ab776a62a78576513ee121424adb19597fbb7613

Signature · 1 line
2.55.0.122.gf85a7e6620