Skip to contentSkip to conversation

What's cooking in git.git (Aug 2023, #01; Wed, 2)

DISCUSSION Junio C Hamano 8 participants 15 messages Aug 2, 2023 – Aug 17, 2023 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

Here are the topics that have been cooking in my tree. Commits prefixed with '+' are in 'next' (being in 'next' is a sign that a topic is stable enough to be used and are candidate to be in a future release). Commits prefixed with '-' are only in 'seen', and aren't considered "accepted" at all and may be annotated with an URL to a message that raises issues but they are no means exhaustive. A topic without enough support may be discarded after a long period of no activity (of course they can be resubmit when new interests arise).

We are getting closer to the final phase of this cycle, which begins when -rc0 preview release is tagged this coming Friday, followed by about 1 1/2 weeks of stabilization period that begins when -rc1 is tagged (cf. tinyurl.com/gitCal). There are a handful of topics that still need reviews before getting merged to 'next', but because the summer in the northern hemisphere is historically a slower season, too few reviewers seem to be active, relative to the number of these topics.

Copies of the source code to Git live in many repositories, and the following is a list of the ones I push into or their mirrors. Some repositories have only a subset of branches.

With maint, master, next, seen, todo:

	git://git.kernel.org/pub/scm/git/git.git/
	git://repo.or.cz/alt-git.git/
	https://kernel.googlesource.com/pub/scm/git/git/
	https://github.com/git/git/
	https://gitlab.com/git-vcs/git/

With all the integration branches and topics broken out:

https://github.com/gitster/git/

Even though the preformatted documentation in HTML and man format are not sources, they are published in these repositories for convenience (replace "htmldocs" with "manpages" for the manual pages):

git://git.kernel.org/pub/scm/git/git-htmldocs.git/
https://github.com/gitster/git-htmldocs.git/

Release tarballs are available at:

https://www.kernel.org/pub/software/scm/git/

--------------------------------------------------
[Graduated to 'master']

* ah/autoconf-fixes (2023-07-19) 3 commits (merged to 'next' on 2023-07-25 at 35ff66e0cb)
+ configure.ac: always save NO_ICONV to config.status
+ configure.ac: don't overwrite NO_CURL option
+ configure.ac: don't overwrite NO_EXPAT option

"./configure --with-expat=no" did not work as a way to refuse use of the expat library on a system with the library installed, which has been corrected.
source: <[email protected]>

* ah/sequencer-rewrite-todo-fix (2023-07-24) 1 commit (merged to 'next' on 2023-07-26 at 24e74d9eda)
+ sequencer: finish parsing the todo list despite an invalid first line

When the user edits "rebase -i" todo file so that it starts with a "fixup", which would make it invalid, the command truncated the
rest of the file before giving an error and returning the control back to the user. Stop truncating to make it easier to correct
such a malformed todo file.
cf. <https://lore.kernel.org/git/[email protected]/>
source: <[email protected]>

* bb/use-trace2-counters-for-fsync-stats (2023-07-20) 1 commit (merged to 'next' on 2023-07-26 at f2c2e3f2b9)
+ wrapper: use trace2 counters to collect fsync stats

Instead of inventing a custom counter variables for debugging, use existing trace2 facility in the fsync customization codepath.
source: <[email protected]>

* jc/tree-walk-drop-base-offset (2023-07-07) 2 commits
(merged to 'next' on 2023-07-25 at cc050c60a6)
+ tree-walk: drop unused base_offset from do_match()
+ tree-walk: lose base_offset that is never used in tree_entry_interesting

Code simplification.
source: <[email protected]>

* ks/ref-filter-describe (2023-07-24) 2 commits (merged to 'next' on 2023-07-26 at f4b3b3b7ef)
+ ref-filter: add new "describe" atom
+ ref-filter: add multiple-option parsing functions

"git branch --list --format=<format>" and friends are taught a new "%(describe)" placeholder.
source: <[email protected]>

--------------------------------------------------
[New Topics]

* bc/ident-dot-is-no-longer-crud-letter (2023-08-02) 1 commit
- ident: don't consider '.' a crud

Exclude "." from the set of characters to be removed from the beginning and the end of the human-readable name.

Will merge to 'next'?
source: <[email protected]>

* jc/unresolve-removal (2023-07-31) 7 commits
- checkout: allow "checkout -m path" to unmerge removed paths
- checkout/restore: add basic tests for --merge
- checkout/restore: refuse unmerging paths unless checking out of the index
- update-index: remove stale fallback code for "--unresolve"
- update-index: use unmerge_index_entry() to support removal
- resolve-undo: allow resurrecting conflicted state that resolved to deletion
- update-index: do not read HEAD and MERGE_HEAD unconditionally

"checkout --merge -- path" and "update-index --unresolve path" did not resurrect conflicted state that was resolved to remove path, but now they do.

Needs review.
source: <[email protected]>

* ew/hash-with-openssl-evp (2023-08-01) 2 commits
- avoid SHA-1 functions deprecated in OpenSSL 3+
- sha256: avoid functions deprecated in OpenSSL 3+

Adjust to OpenSSL 3+, which deprecates its SHA-1 functions based on its traditional API, by using its EVP API instead.

Will merge to 'next'.
source: <[email protected]>

* rj/status-bisect-while-rebase (2023-08-01) 1 commit
- status: fix branch shown when not only bisecting

"git status" is taught to show both the branch being bisected and being rebased when both are in effect at the same time.

Needs review.
source: <[email protected]>

--------------------------------------------------
[Stalled]

* tk/cherry-pick-sequence-requires-clean-worktree (2023-06-01) 1 commit
- cherry-pick: refuse cherry-pick sequence if index is dirty

"git cherry-pick A" that replays a single commit stopped before clobbering local modification, but "git cherry-pick A..B" did not, which has been corrected.

Expecting a reroll.
cf. <[email protected]>
source: <[email protected]>

* ab/tag-object-type-errors (2023-05-10) 4 commits
- tag: don't emit potentially incorrect "object is a X, not a Y"
- tag: don't misreport type of tagged objects in errors
- object tests: add test for unexpected objects in tags
- Merge branch 'jk/parse-object-type-mismatch' into ab/tag-object-type-errors

Hardening checks around mismatched object types when one of those objects is a tag.

Will discard.
Stalled for too long.
source: <[email protected]>

* ob/revert-of-revert (2023-05-05) 1 commit
- sequencer: beautify subject of reverts of reverts

Instead of "Revert "Revert "original"", give "Reapply "original"" as the title for a revert of a revert.

Will discard.
Have been expecting a hopefully final reroll for too long. Looking much better, except for minor cosmetic issues. cf. <[email protected]>
source: <[email protected]>

* pw/rebase-i-after-failure (2023-08-01) 7 commits
- rebase -i: fix adding failed command to the todo list
- rebase --continue: refuse to commit after failed command
- rebase: fix rewritten list for failed pick
- sequencer: factor out part of pick_commits()
- sequencer: use rebase_path_message()
- rebase -i: remove patch file after conflict resolution
- rebase -i: move unlink() calls

Various fixes to the behaviour of "rebase -i" when the command got interrupted by conflicting changes.

Will merge to 'next'?
cf. <[email protected]>
cf. <[email protected]>
source: <[email protected]>

--------------------------------------------------
[Cooking]

* ew/sha256-gcrypt-leak-fixes (2023-07-31) 3 commits (merged to 'next' on 2023-08-01 at eed83801c3)
+ sha256/gcrypt: die on gcry_md_open failures
+ sha256/gcrypt: fix memory leak with SHA-256 repos
+ sha256/gcrypt: fix build with SANITIZE=leak

Leakfixes.

Will merge to 'master'.
source: <[email protected]>

* rs/bundle-parseopt-cleanup (2023-07-31) 1 commit (merged to 'next' on 2023-08-01 at 405eb138fa)
+ bundle: use OPT_PASSTHRU_ARGV

Code clean-up.

Will merge to 'master'.
source: <[email protected]>

* pv/doc-submodule-update-settings (2023-07-25) 1 commit (merged to 'next' on 2023-07-27 at e27b5b7ba8)
+ doc: highlight that .gitmodules does not support !command

Rewrite the description of giving a custom command to the submodule.<name>.update configuraiton variable.

Will merge to 'master'.
source: <[email protected]>

* la/doc-choose-starting-point-fixup (2023-07-27) 3 commits (merged to 'next' on 2023-07-28 at 047dcae31c)
+ SubmittingPatches: use of older maintenance tracks is an exception
+ SubmittingPatches: explain why 'next' and above are inappropriate base
+ SubmittingPatches: choice of base for fixing an older maintenance track (this branch uses la/doc-choose-starting-point.)

Clarify how to pick a starting point for a new topic in the SubmittingPatches document.

Will merge to 'master', together with the underlying topic.
source: <[email protected]>
source: <[email protected]>

* am/doc-sha256 (2023-07-31) 1 commit
(merged to 'next' on 2023-08-01 at d7419bf527)
+ doc: sha256 is no longer experimental

Tone down the warning on SHA-256 repositories being an experimental curiosity. We do not have support for them to interoperate with traditional SHA-1 repositories, but at this point, we do not plan to make breaking changes to SHA-256 repositories and there is no longer need for such a strongly phrased warning.

Will merge to 'master'.
source: <ZMe6KmzZGVubYpvO@adams>

* hy/blame-in-bare-with-contents (2023-07-21) 1 commit (merged to 'next' on 2023-07-31 at 39ac96d8d8)
+ blame: allow --contents to work with bare repo

"git blame --contents=file" has been taught to work in a bare repository.

Will merge to 'master'.
source: <[email protected]>

* ja/worktree-orphan-fix (2023-07-26) 3 commits (merged to 'next' on 2023-07-27 at e475016065)
+ t2400: rewrite regex to avoid unintentional PCRE
+ builtin/worktree.c: convert tab in advice to space
+ t2400: drop no-op --sq from rev-parse call

Fix tests with unportable regex patterns.

Will merge to 'master'.
source: <[email protected]>

* jc/retire-get-sha1-hex (2023-07-24) 1 commit
(merged to 'next' on 2023-07-27 at eeb9cc37f5)
+ hex: retire get_sha1_hex()

The implementation of "get_sha1_hex()" that reads a hexadecimal string that spells a full object name has been extended to cope with any hash function used in the repository, but the "sha1" in its name survived. Rename it to get_hash_hex(), a name that is more consistent within its friends like get_hash_hex_algop().

Will merge to 'master'.
source: <[email protected]>

* rs/parse-options-negation-help (2023-07-24) 5 commits
- parse-options: show negatability of options in short help
- t1502: test option negation
- t1502: move optionspec help output to a file
- t1502, docs: disallow --no-help
- subtree: disallow --no-{help,quiet,debug,branch,message}

"git cmd -h" learned to signal which options can be negated by listing such options like "--[no-]opt".

Comments?
Would showing "--[[no-]no-]opt" for "no-opt" be worth it? cf. <[email protected]>
source: <[email protected]>

* tb/commit-graph-tests (2023-07-24) 5 commits
(merged to 'next' on 2023-07-31 at 740a260315)
+ t/lib-commit-graph.sh: avoid sub-shell in graph_git_behavior()
+ t5328: avoid top-level directory changes
+ t5318: avoid top-level directory changes
+ t/lib-commit-graph.sh: avoid directory change in graph_git_behavior()
+ t/lib-commit-graph.sh: allow graph_read_expect() in sub-directories

Test updates.

Will merge to 'master'.
source: <[email protected]>

* la/doc-choose-starting-point (2023-07-14) 5 commits
(merged to 'next' on 2023-07-19 at 5a807cae46)
+ SubmittingPatches: simplify guidance for choosing a starting point
+ SubmittingPatches: emphasize need to communicate non-default starting points
+ SubmittingPatches: de-emphasize branches as starting points
+ SubmittingPatches: discuss subsystems separately from git.git
+ SubmittingPatches: reword awkward phrasing
(this branch is used by la/doc-choose-starting-point-fixup.)

Clarify how to choose the starting point for a new topic in developer guidance document.

Will merge to 'master' together with the follow-on topic.
source: <[email protected]>

* jc/doc-sent-patch-now-what (2023-07-27) 1 commit (merged to 'next' on 2023-07-31 at 51f5d9d465)
+ MyFirstContribution: refrain from self-iterating too much

Process document update.

Will merge to 'master'.
source: <[email protected]>

* jc/parse-options-short-help (2023-07-19) 3 commits (merged to 'next' on 2023-07-31 at e076d1f497)
+ short help: allow a gap smaller than USAGE_GAP
+ remote: simplify "remote add --tags" help text
+ short help: allow multi-line opthelp

Command line parser fix, and a small parse-options API update.

Will merge to 'master'.
source: <[email protected]>

* sl/sparse-check-attr (2023-07-18) 3 commits
- check-attr: integrate with sparse-index
- attr.c: read attributes in a sparse directory
- t1092: add tests for 'git check-attr'

Teach "git check-attr" work better with sparse-index.

Expecting a reroll.
cf. <[email protected]> cf. <[email protected]>
source: <[email protected]>

* jc/branch-in-use-error-message (2023-07-21) 1 commit (merged to 'next' on 2023-07-31 at 22f17d131b)
+ branch: update the message to refuse touching a branch in-use

"git branch -f X" to repoint the branch X seid that X was "checked out" in another worktree, even when branch X was not and instead being bisected or rebased. The message was reworded to say the branch was "in use".

Will merge to 'master'.
source: <[email protected]>

* mh/credential-erase-improvements-more (2023-07-26) 2 commits
- credential/wincred: erase matching creds only
- credential/libsecret: erase matching creds only

Update two credential helpers to correctly match which credential to erase; they dropped not the ones with stale password.

Needs review.
source: <[email protected]>
source: <[email protected]>

* cc/repack-sift-filtered-objects-to-separate-pack (2023-07-24) 8 commits . gc: add gc.repackFilterTo config option
. repack: implement --filter-to for storing filtered out objects . gc: add gc.repackFilter config option
. repack: add --filter=<filter-spec> option
. repack: refactor finding pack prefix
. repack: refactor finishing pack-objects command
. t/helper: add 'find-pack' test-tool
. pack-objects: allow --filter without --stdout

"git repack" machinery learns to pay attention to the "--filter=" option.

Breaks CI with some environment variables configured. cf. <[email protected]>
source: <[email protected]>

* js/doc-unit-tests (2023-06-30) 1 commit
- unit tests: Add a project plan document

Process to add some form of low-level unit tests has started.

Still filling in blanks.
source: <0169ce6fb9ccafc089b74ae406db0d1a8ff8ac65.1688165272.git.steadmon@google.com>

* jt/path-filter-fix (2023-08-01) 7 commits
- commit-graph: new filter ver. that fixes murmur3
- repo-settings: introduce commitgraph.changedPathsVersion
- t4216: test changed path filters with high bit paths
- t/helper/test-read-graph: implement bloom-filters mode
- bloom.h: make load_bloom_filter_from_graph() public
- t/helper/test-read-graph.c: extract dump_graph_info()
- gitformat-commit-graph: describe version 2 of BDAT

The Bloom filter used for path limited history traversal was broken on systems whose "char" is unsigned; update the implementation and bump the format version to 2.

Still under discussion.
cf. <[email protected]>
source: <[email protected]>

* mh/credential-libsecret-attrs (2023-06-16) 1 commit
- credential/libsecret: store new attributes

The way authentication related data other than passwords (e.g. oath token and password expiration data) are stored in libsecret keyrings has been rethought.

Needs review.
source: <[email protected]>

* cc/git-replay (2023-06-03) 15 commits
- replay: stop assuming replayed branches do not diverge
- replay: add --contained to rebase contained branches
- replay: add --advance or 'cherry-pick' mode
- replay: disallow revision specific options and pathspecs
- replay: use standard revision ranges
- replay: make it a minimal server side command
- replay: remove HEAD related sanity check
- replay: remove progress and info output
- replay: add an important FIXME comment about gpg signing
- replay: don't simplify history
- replay: introduce pick_regular_commit()
- replay: die() instead of failing assert()
- replay: start using parse_options API
- replay: introduce new builtin
- t6429: remove switching aspects of fast-rebase

What's the status of this thing?
source: <[email protected]>

--------------------------------------------------
[Discarded]

* jc/doc-submodule-update-settings (2023-07-13) 1 commit . submodule: clarify that "!custom command" is the only oddball

Rewrite the description of giving a custom command to the submodule.<name>.update configuraiton variable.

Superseded by pv/doc-submodule-update-settings topic.
source: <[email protected]>

* jc/rerere-read-rr-fix (2023-07-21) 1 commit . rerere: match the hash algorithm with its length

SHA-256 fix.

Superseded by jc/retire-get-sha1-hex
source: <[email protected]>

* cb/checkout-same-branch-twice (2023-03-22) 2 commits
. SQUASH??? the test marked to expect failure passes from day one . checkout/switch: disallow checking out same branch in multiple worktrees

"git checkout -B $branch" failed to protect against checking out a branch that is checked out elsewhere, unlike "git branch -f" did.

Have been expecting a hopefully minor and final reroll for too long. cf. <CAPUEspj_Bh+LgYLnWfeBdcq_uV5Cbou-7H51GLFjzSa5Qzby9w@mail.gmail.com>
source: <[email protected]>

* ed/fsmonitor-windows-named-pipe (2023-03-24) 1 commit
. fsmonitor: handle differences between Windows named pipe functions

Fix fsmonitor on Windows when the filesystem path contains certain characters.

Have been expecting a reroll for too long.
cf. <[email protected]>
source: <[email protected]>

* rn/sparse-diff-index (2023-04-10) 1 commit
. diff-index: enable sparse index

"git diff-index" command has been taught to work better with the sparse index.

Have been expecting a reroll for too long.
cf. <[email protected]>
source: <[email protected]>

* es/recurse-submodules-option-is-a-bool (2023-04-10) 1 commit . usage: clarify --recurse-submodules as a boolean

The "--[no-]recurse-submodules" option of "git checkout" and others supported an undocumented syntax --recurse-submodules=<value> where the value can spell a Boolean in various ways. The support for the syntax is being dropped.

Have been expecting a reroll for too long.
cf. <ZDSTFwMFO7vbj/[email protected]>
source: <ZDSTFwMFO7vbj/[email protected]>

* jc/checkout-merge-fix (2023-07-28) 2 commits
. checkout/restore: add basic tests for --merge
. checkout/restore: refuse unmerging paths unless checking out of the index

"git checkout/restore --merge -- $path" improvements.

Superseded by jc/unresolve-removal
source: <[email protected]>

* jc/resolve-undo-fixes (2023-07-28) 4 commits
. update-index: remove stale fallback code for "--unresolve"
. update-index: use unmerge_index_entry() to support removal
. resolve-undo: allow resurrecting conflicted state that resolved to deletion . update-index: do not read HEAD and MERGE_HEAD unconditionally

Assorted fixes and clean-up around resolve-undo data.

Superseded by jc/unresolve-removal
source: <[email protected]>

#1 lore raw

jt/path-filter-fix (was: Re: What's cooking in git.git (Aug 2023, #01; Wed, 2))

Quoted 9 lines · Junio C Hamano

* jt/path-filter-fix (2023-08-01) 7 commits
- commit-graph: new filter ver. that fixes murmur3
- repo-settings: introduce commitgraph.changedPathsVersion
- t4216: test changed path filters with high bit paths
- t/helper/test-read-graph: implement bloom-filters mode
- bloom.h: make load_bloom_filter_from_graph() public
- t/helper/test-read-graph.c: extract dump_graph_info()
- gitformat-commit-graph: describe version 2 of BDAT

Quoted 7 lines

The Bloom filter used for path limited history traversal was broken on systems whose "char" is unsigned; update the implementation and bump the format version to 2.

Still under discussion.
cf. <[email protected]>
source: <[email protected]>

I am happy with the most recent round, but I think that it is probably a little late in the cycle to be merging down such a large change.

I wouldn't be opposed if you did so, but it may be worth waiting until we're on the other side of 2.42 so that other reviewers have a chance to look at this one as well.

Thanks,
Taylor

#2 lore raw

Re: jt/path-filter-fix

Quoted 18 lines · Taylor Blau
Quoted 9 lines · Junio C Hamano

* jt/path-filter-fix (2023-08-01) 7 commits
- commit-graph: new filter ver. that fixes murmur3
- repo-settings: introduce commitgraph.changedPathsVersion
- t4216: test changed path filters with high bit paths
- t/helper/test-read-graph: implement bloom-filters mode
- bloom.h: make load_bloom_filter_from_graph() public
- t/helper/test-read-graph.c: extract dump_graph_info()
- gitformat-commit-graph: describe version 2 of BDAT

Quoted 7 lines

The Bloom filter used for path limited history traversal was broken on systems whose "char" is unsigned; update the implementation and bump the format version to 2.

Still under discussion.
cf. <[email protected]>
source: <[email protected]>

Quoted 5 lines

I am happy with the most recent round, but I think that it is probably a little late in the cycle to be merging down such a large change.

I wouldn't be opposed if you did so, but it may be worth waiting until we're on the other side of 2.42 so that other reviewers have a chance to

Since I hear some folks (not just Google) base their own edition of Git on 'next', it probably is a good idea to merge it to 'next' and have their users help find potential issues in it, as I agree that the area it touches is important in the correctness department. Of course, it is important enough that the topic may very well want to be cooked longer than the usual "for at least one calendar week" in 'next', so I tend to agree that in a first few batches after the release may be the best time to have it graduate (if it turns out to be OK).

Thanks.

#3 lore raw

Re: jt/path-filter-fix

Quoted 26 lines · Junio C Hamano
Quoted 18 lines · Taylor Blau
Quoted 9 lines · Junio C Hamano

* jt/path-filter-fix (2023-08-01) 7 commits
- commit-graph: new filter ver. that fixes murmur3
- repo-settings: introduce commitgraph.changedPathsVersion
- t4216: test changed path filters with high bit paths
- t/helper/test-read-graph: implement bloom-filters mode
- bloom.h: make load_bloom_filter_from_graph() public
- t/helper/test-read-graph.c: extract dump_graph_info()
- gitformat-commit-graph: describe version 2 of BDAT

Quoted 7 lines

The Bloom filter used for path limited history traversal was broken on systems whose "char" is unsigned; update the implementation and bump the format version to 2.

Still under discussion.
cf. <[email protected]>
source: <[email protected]>

Quoted 5 lines

I am happy with the most recent round, but I think that it is probably a little late in the cycle to be merging down such a large change.

I wouldn't be opposed if you did so, but it may be worth waiting until we're on the other side of 2.42 so that other reviewers have a chance to

Quoted 9 lines

Since I hear some folks (not just Google) base their own edition of Git on 'next', it probably is a good idea to merge it to 'next' and have their users help find potential issues in it, as I agree that the area it touches is important in the correctness department. Of course, it is important enough that the topic may very well want to be cooked longer than the usual "for at least one calendar week" in 'next', so I tend to agree that in a first few batches after the release may be the best time to have it graduate (if it turns out to be OK).

Yep, I agree with all of that. Thanks for juggling the merges, as always :-).

Thanks,
Taylor

#5 lore raw
In reply to a message not in the archive

(Sorry I somehow forgot to put the mailing list in Cc: of this.)

Quoted 50 lines · Christian Couder
Quoted 10 lines · Junio C Hamano

* cc/repack-sift-filtered-objects-to-separate-pack (2023-07-24) 8 commits . gc: add gc.repackFilterTo config option
. repack: implement --filter-to for storing filtered out objects . gc: add gc.repackFilter config option
. repack: add --filter=<filter-spec> option
. repack: refactor finding pack prefix
. repack: refactor finishing pack-objects command
. t/helper: add 'find-pack' test-tool
. pack-objects: allow --filter without --stdout

Quoted 6 lines

"git repack" machinery learns to pay attention to the "--filter=" option.

Breaks CI with some environment variables configured. cf. <[email protected]>
source: <[email protected]>

I am working on this and hope to send a version 4 soon.

Quoted 17 lines

* cc/git-replay (2023-06-03) 15 commits
- replay: stop assuming replayed branches do not diverge
- replay: add --contained to rebase contained branches
- replay: add --advance or 'cherry-pick' mode
- replay: disallow revision specific options and pathspecs
- replay: use standard revision ranges
- replay: make it a minimal server side command
- replay: remove HEAD related sanity check
- replay: remove progress and info output
- replay: add an important FIXME comment about gpg signing
- replay: don't simplify history
- replay: introduce pick_regular_commit()
- replay: die() instead of failing assert()
- replay: start using parse_options API
- replay: introduce new builtin
- t6429: remove switching aspects of fast-rebase

Quoted 2 lines

What's the status of this thing?
source: <[email protected]>

I have a 2 week long vacation starting soon, so I will likely not have time to work on a new round in the next 3 weeks.

However in a recent article
(https://github.blog/2023-07-27-scaling-merge-ort-across-github/)
GitHub says that they are already using git-replay (though not sure it's this exact version or another one), and GitLab plans to use it too. So I guess it's worth keeping.

#6 lore raw
Quoted 1 line · Christian Couder

(Sorry I somehow forgot to put the mailing list in Cc: of this.)

This seems to be a response to an older issue of the report, but I take it that the status of these two topics haven't changed since then.

Quoted 7 lines
Quoted 7 lines
Quoted 5 lines

* cc/repack-sift-filtered-objects-to-separate-pack (2023-07-24) 8 commits
...
Breaks CI with some environment variables configured.
cf. <[email protected]>
source: <[email protected]>

I am working on this and hope to send a version 4 soon.

Sounds good. Thanks.

Quoted 8 lines
Quoted 8 lines
Quoted 5 lines

* cc/git-replay (2023-06-03) 15 commits
...

What's the status of this thing?
source: <[email protected]>

I have a 2 week long vacation starting soon, so I will likely not have time to work on a new round in the next 3 weeks.

OK. Enjoy.

Quoted 5 lines
Quoted 5 lines

However in a recent article
(https://github.blog/2023-07-27-scaling-merge-ort-across-github/)
GitHub says that they are already using git-replay (though not sure it's this exact version or another one), and GitLab plans to use it too.

So there are plenty folks who are capable of reviewing but they are not interested in giving it to the general public by withholding their reviews ;-)?

#7 lore raw
Quoted 9 lines · Junio C Hamano
Quoted 5 lines
Quoted 5 lines

However in a recent article
(https://github.blog/2023-07-27-scaling-merge-ort-across-github/)
GitHub says that they are already using git-replay (though not sure it's this exact version or another one), and GitLab plans to use it too.

So there are plenty folks who are capable of reviewing but they are not interested in giving it to the general public by withholding their reviews ;-)?

I think the most capable reviewer would be Johannes, who was most involved the Git-related parts of this work at GitHub.

I was party to most of the changes that went through in this area, but didn't look closely.

Thanks,
Taylor

#8 lore raw
In reply to Junio C Hamano
Quoted 1 line · Junio C Hamano

[...]

Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

However in a recent article
(https://github.blog/2023-07-27-scaling-merge-ort-across-github/)
GitHub says that they are already using git-replay (though not sure it's this exact version or another one), and GitLab plans to use it too.

So there are plenty folks who are capable of reviewing but they are not interested in giving it to the general public by withholding their reviews ;-)?

I can see how one could jump to that conclusion, but I don't think it's warranted. I have a little information that might shed some light:

-----

At the beginning of July, well before the above link came out, Johannes sent me an email pointing me at
https://github.blog/changelog/2023-06-28-rebase-commits-now-created-using-the-merge-ort-strategy/.
In the email, he also noted my earlier stated concerns on the list (about releasing replay early possibly painting us into a corner preventing some desired goals for the tool from being realized), and tried to ensure we could still work towards having a replay command like the one I envisioned while also asking for my thoughts on pushing for the current portion of work to be published and included in git. My sense was he was pushing for the work to be released, but just being careful about my concerns first.

Unfortunately, I was on vacation that week, and sadly have otherwise been buried in coming up to speed on a new team and haven't had time to even respond to git-related stuff. I didn't respond to him until late last week.

I pointed out that the 'EXPERIMENTAL' banner addresses most of my concerns so it should be fine to move forward, but I suspect my delay has resulted in him now being buried in other matters, and in the mean-time the article Christian highlighted was produced by others.

-----

Anyway, hope that helps. I did review V2 and have been meaning to comment on V3 (and respond to Toon's comments), though not sure my "review" counts as such since I'm the original author of most of the patches...

(And sorry for being missing in action. I've flagged a few other emails that I'm hoping I can follow up on, but my time is currently quite limited...)

#10 lore raw
In reply to Junio C Hamano
Quoted 5 lines · Junio C Hamano

* rj/status-bisect-while-rebase (2023-08-01) 1 commit
- status: fix branch shown when not only bisecting

"git status" is taught to show both the branch being bisected and being rebased when both are in effect at the same time.

Maybe this suggest that something has been added, but only fixed.

Perhaps we can say something like:

"git status" was not showing accurate information when bisect and other operation are in effect at the same time.

Quoted 3 lines

Needs review.
source: <[email protected]>

Thanks.

#11 lore raw

leak in jt/path-filter-fix, was Re: What's cooking in git.git (Aug 2023, #01; Wed, 2)

In reply to Junio C Hamano
Quoted 9 lines · Junio C Hamano

* jt/path-filter-fix (2023-08-01) 7 commits
- commit-graph: new filter ver. that fixes murmur3
- repo-settings: introduce commitgraph.changedPathsVersion
- t4216: test changed path filters with high bit paths
- t/helper/test-read-graph: implement bloom-filters mode
- bloom.h: make load_bloom_filter_from_graph() public
- t/helper/test-read-graph.c: extract dump_graph_info()
- gitformat-commit-graph: describe version 2 of BDAT

Quoted 7 lines

The Bloom filter used for path limited history traversal was broken on systems whose "char" is unsigned; update the implementation and bump the format version to 2.

Still under discussion.
cf. <[email protected]>
source: <[email protected]>

Since this hit 'next', Coverity complained about a small leak. Fixed by the patch below.

-- >8 --
Subject: [PATCH] commit-graph: fix small leak with invalid changedPathsVersion

When writing a commit graph, we sanity-check the value of commitgraph.changedPathsVersion and return early if it's invalid. But we only do so after allocating the write_commit_graph_context, meaing we leak it. We could "goto cleanup" to fix this, but instead let's push this check to the top of the function with the other early checks which return before doing any work.

---
 commit-graph.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
commit-graph.c +6 -6
index 38edb12669..ccc21c6708 100644
@@ -2372,6 +2372,12 @@ int write_commit_graph(struct object_directory *odb,
 	}
 	if (!commit_graph_compatible(r))
 		return 0;
+	if (r->settings.commit_graph_changed_paths_version < -1
+	    || r->settings.commit_graph_changed_paths_version > 2) {
+		warning(_("attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported"),
+			r->settings.commit_graph_changed_paths_version);
+		return 0;
+	}
 
 	CALLOC_ARRAY(ctx, 1);
 	ctx->r = r;
@@ -2384,12 +2390,6 @@ int write_commit_graph(struct object_directory *odb,
 	ctx->write_generation_data = (get_configured_generation_version(r) == 2);
 	ctx->num_generation_data_overflows = 0;
 
-	if (r->settings.commit_graph_changed_paths_version < -1
-	    || r->settings.commit_graph_changed_paths_version > 2) {
-		warning(_("attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported"),
-			r->settings.commit_graph_changed_paths_version);
-		return 0;
-	}
 	bloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2
 		? 2 : 1;
 	bloom_settings.bits_per_entry = git_env_ulong("GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY",
Signature · 1 line
2.42.0.rc0.376.g66bfc4f195
#12 lore raw

Re: leak in jt/path-filter-fix, was Re: What's cooking in git.git (Aug 2023, #01; Wed, 2)

Quoted 8 lines · Jeff King

Subject: [PATCH] commit-graph: fix small leak with invalid changedPathsVersion

When writing a commit graph, we sanity-check the value of commitgraph.changedPathsVersion and return early if it's invalid. But we only do so after allocating the write_commit_graph_context, meaing we leak it. We could "goto cleanup" to fix this, but instead let's push this check to the top of the function with the other early checks which return before doing any work.

I am tempted to do this on top, but maybe it is just churn. I dunno.

-- >8 --
Subject: [PATCH] commit-graph: store write context on stack rather than heap

In write_commit_graph(), we allocate our context struct on the heap. But there's no reason to do so, since it's small and we always free it when exiting the function. Converting it to a stack variable is slightly more efficient, but also can prevent some leaks (such as the one fixed in the previous commit, though to be fair this only helps for such a limited leak; once any sub-fields of the struct require allocation, it has to be cleaned up).

---
 commit-graph.c | 140 ++++++++++++++++++++++++-------------------------
 1 file changed, 69 insertions(+), 71 deletions(-)
commit-graph.c +69 -71
index ccc21c6708..b8db056f37 100644
@@ -2358,7 +2358,7 @@ int write_commit_graph(struct object_directory *odb,
 		       const struct commit_graph_opts *opts)
 {
 	struct repository *r = the_repository;
-	struct write_commit_graph_context *ctx;
+	struct write_commit_graph_context ctx;
 	uint32_t i;
 	int res = 0;
 	int replace = 0;
@@ -2379,16 +2379,16 @@ int write_commit_graph(struct object_directory *odb,
 		return 0;
 	}
 
-	CALLOC_ARRAY(ctx, 1);
-	ctx->r = r;
-	ctx->odb = odb;
-	ctx->append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;
-	ctx->report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;
-	ctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;
-	ctx->opts = opts;
-	ctx->total_bloom_filter_data_size = 0;
-	ctx->write_generation_data = (get_configured_generation_version(r) == 2);
-	ctx->num_generation_data_overflows = 0;
+	memset(&ctx, 0, sizeof(ctx));
+	ctx.r = r;
+	ctx.odb = odb;
+	ctx.append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;
+	ctx.report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;
+	ctx.split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;
+	ctx.opts = opts;
+	ctx.total_bloom_filter_data_size = 0;
+	ctx.write_generation_data = (get_configured_generation_version(r) == 2);
+	ctx.num_generation_data_overflows = 0;
 
 	bloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2
 		? 2 : 1;
@@ -2398,14 +2398,14 @@ int write_commit_graph(struct object_directory *odb,
 						  bloom_settings.num_hashes);
 	bloom_settings.max_changed_paths = git_env_ulong("GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS",
 							 bloom_settings.max_changed_paths);
-	ctx->bloom_settings = &bloom_settings;
+	ctx.bloom_settings = &bloom_settings;
 
 	init_topo_level_slab(&topo_levels);
-	ctx->topo_levels = &topo_levels;
+	ctx.topo_levels = &topo_levels;
 
-	prepare_commit_graph(ctx->r);
-	if (ctx->r->objects->commit_graph) {
-		struct commit_graph *g = ctx->r->objects->commit_graph;
+	prepare_commit_graph(ctx.r);
+	if (ctx.r->objects->commit_graph) {
+		struct commit_graph *g = ctx.r->objects->commit_graph;
 
 		while (g) {
 			g->topo_levels = &topo_levels;
@@ -2414,128 +2414,126 @@ int write_commit_graph(struct object_directory *odb,
 	}
 
 	if (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)
-		ctx->changed_paths = 1;
+		ctx.changed_paths = 1;
 	if (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {
 		struct commit_graph *g;
 
-		g = ctx->r->objects->commit_graph;
+		g = ctx.r->objects->commit_graph;
 
 		/* We have changed-paths already. Keep them in the next graph */
 		if (g && g->bloom_filter_settings) {
-			ctx->changed_paths = 1;
-			ctx->bloom_settings = g->bloom_filter_settings;
+			ctx.changed_paths = 1;
+			ctx.bloom_settings = g->bloom_filter_settings;
 		}
 	}
 
-	if (ctx->split) {
-		struct commit_graph *g = ctx->r->objects->commit_graph;
+	if (ctx.split) {
+		struct commit_graph *g = ctx.r->objects->commit_graph;
 
 		while (g) {
-			ctx->num_commit_graphs_before++;
+			ctx.num_commit_graphs_before++;
 			g = g->base_graph;
 		}
 
-		if (ctx->num_commit_graphs_before) {
-			ALLOC_ARRAY(ctx->commit_graph_filenames_before, ctx->num_commit_graphs_before);
-			i = ctx->num_commit_graphs_before;
-			g = ctx->r->objects->commit_graph;
+		if (ctx.num_commit_graphs_before) {
+			ALLOC_ARRAY(ctx.commit_graph_filenames_before, ctx.num_commit_graphs_before);
+			i = ctx.num_commit_graphs_before;
+			g = ctx.r->objects->commit_graph;
 
 			while (g) {
-				ctx->commit_graph_filenames_before[--i] = xstrdup(g->filename);
+				ctx.commit_graph_filenames_before[--i] = xstrdup(g->filename);
 				g = g->base_graph;
 			}
 		}
 
-		if (ctx->opts)
-			replace = ctx->opts->split_flags & COMMIT_GRAPH_SPLIT_REPLACE;
+		if (ctx.opts)
+			replace = ctx.opts->split_flags & COMMIT_GRAPH_SPLIT_REPLACE;
 	}
 
-	ctx->approx_nr_objects = repo_approximate_object_count(the_repository);
+	ctx.approx_nr_objects = repo_approximate_object_count(the_repository);
 
-	if (ctx->append && ctx->r->objects->commit_graph) {
-		struct commit_graph *g = ctx->r->objects->commit_graph;
+	if (ctx.append && ctx.r->objects->commit_graph) {
+		struct commit_graph *g = ctx.r->objects->commit_graph;
 		for (i = 0; i < g->num_commits; i++) {
 			struct object_id oid;
 			oidread(&oid, g->chunk_oid_lookup + g->hash_len * i);
-			oid_array_append(&ctx->oids, &oid);
+			oid_array_append(&ctx.oids, &oid);
 		}
 	}
 
 	if (pack_indexes) {
-		ctx->order_by_pack = 1;
-		if ((res = fill_oids_from_packs(ctx, pack_indexes)))
+		ctx.order_by_pack = 1;
+		if ((res = fill_oids_from_packs(&ctx, pack_indexes)))
 			goto cleanup;
 	}
 
 	if (commits) {
-		if ((res = fill_oids_from_commits(ctx, commits)))
+		if ((res = fill_oids_from_commits(&ctx, commits)))
 			goto cleanup;
 	}
 
 	if (!pack_indexes && !commits) {
-		ctx->order_by_pack = 1;
-		fill_oids_from_all_packs(ctx);
+		ctx.order_by_pack = 1;
+		fill_oids_from_all_packs(&ctx);
 	}
 
-	close_reachable(ctx);
+	close_reachable(&ctx);
 
-	copy_oids_to_commits(ctx);
+	copy_oids_to_commits(&ctx);
 
-	if (ctx->commits.nr >= GRAPH_EDGE_LAST_MASK) {
+	if (ctx.commits.nr >= GRAPH_EDGE_LAST_MASK) {
 		error(_("too many commits to write graph"));
 		res = -1;
 		goto cleanup;
 	}
 
-	if (!ctx->commits.nr && !replace)
+	if (!ctx.commits.nr && !replace)
 		goto cleanup;
 
-	if (ctx->split) {
-		split_graph_merge_strategy(ctx);
+	if (ctx.split) {
+		split_graph_merge_strategy(&ctx);
 
 		if (!replace)
-			merge_commit_graphs(ctx);
+			merge_commit_graphs(&ctx);
 	} else
-		ctx->num_commit_graphs_after = 1;
+		ctx.num_commit_graphs_after = 1;
 
-	ctx->trust_generation_numbers = validate_mixed_generation_chain(ctx->r->objects->commit_graph);
+	ctx.trust_generation_numbers = validate_mixed_generation_chain(ctx.r->objects->commit_graph);
 
-	compute_topological_levels(ctx);
-	if (ctx->write_generation_data)
-		compute_generation_numbers(ctx);
+	compute_topological_levels(&ctx);
+	if (ctx.write_generation_data)
+		compute_generation_numbers(&ctx);
 
-	if (ctx->changed_paths)
-		compute_bloom_filters(ctx);
+	if (ctx.changed_paths)
+		compute_bloom_filters(&ctx);
 
-	res = write_commit_graph_file(ctx);
+	res = write_commit_graph_file(&ctx);
 
-	if (ctx->split)
-		mark_commit_graphs(ctx);
+	if (ctx.split)
+		mark_commit_graphs(&ctx);
 
-	expire_commit_graphs(ctx);
+	expire_commit_graphs(&ctx);
 
 cleanup:
-	free(ctx->graph_name);
-	free(ctx->commits.list);
-	oid_array_clear(&ctx->oids);
+	free(ctx.graph_name);
+	free(ctx.commits.list);
+	oid_array_clear(&ctx.oids);
 	clear_topo_level_slab(&topo_levels);
 
-	if (ctx->commit_graph_filenames_after) {
-		for (i = 0; i < ctx->num_commit_graphs_after; i++) {
-			free(ctx->commit_graph_filenames_after[i]);
-			free(ctx->commit_graph_hash_after[i]);
+	if (ctx.commit_graph_filenames_after) {
+		for (i = 0; i < ctx.num_commit_graphs_after; i++) {
+			free(ctx.commit_graph_filenames_after[i]);
+			free(ctx.commit_graph_hash_after[i]);
 		}
 
-		for (i = 0; i < ctx->num_commit_graphs_before; i++)
-			free(ctx->commit_graph_filenames_before[i]);
+		for (i = 0; i < ctx.num_commit_graphs_before; i++)
+			free(ctx.commit_graph_filenames_before[i]);
 
-		free(ctx->commit_graph_filenames_after);
-		free(ctx->commit_graph_filenames_before);
-		free(ctx->commit_graph_hash_after);
+		free(ctx.commit_graph_filenames_after);
+		free(ctx.commit_graph_filenames_before);
+		free(ctx.commit_graph_hash_after);
 	}
 
-	free(ctx);
-
 	return res;
 }
 
Signature · 1 line
2.42.0.rc0.376.g66bfc4f195
#13 lore raw

Re: leak in jt/path-filter-fix, was Re: What's cooking in git.git (Aug 2023, #01; Wed, 2)

In reply to Jeff King
Quoted 2 lines · Jeff King

Since this hit 'next', Coverity complained about a small leak. Fixed by the patch below.

The patch below looks good and makes sense to me. That warning is awfully long, though ;-).

In any event, I expect that 'next' will be rewound before this topic graduates, since it is meaty and we are in the middle of the -rc phase.

...and we also have my series on top [1], so it may be worthwhile for you, Jonathan, and I to figure out how to combine our efforts here. I think that this could likely get squashed in to Jonathan's topic if/after it gets ejected from 'next'. We can take my patches together in the same series, separately in a different one, or discard them altogether.

If we do decide to pursue the approach in [1], I think combining everything together into one big series makes the most sense for ease of merging.

[1]: https://lore.kernel.org/git/[email protected]/

Thanks,
Taylor

#14 lore raw

Re: leak in jt/path-filter-fix, was Re: What's cooking in git.git (Aug 2023, #01; Wed, 2)

Quoted 17 lines · Taylor Blau
Quoted 2 lines · Jeff King

Since this hit 'next', Coverity complained about a small leak. Fixed by the patch below.

The patch below looks good and makes sense to me. That warning is awfully long, though ;-).

In any event, I expect that 'next' will be rewound before this topic graduates, since it is meaty and we are in the middle of the -rc phase.

...and we also have my series on top [1], so it may be worthwhile for you, Jonathan, and I to figure out how to combine our efforts here. I think that this could likely get squashed in to Jonathan's topic if/after it gets ejected from 'next'. We can take my patches together in the same series, separately in a different one, or discard them altogether.

Quoted 3 lines

If we do decide to pursue the approach in [1], I think combining everything together into one big series makes the most sense for ease of merging.

Yeah, I am just coming back from vacation and didn't follow all of the discussion on this topic carefully. The few patches I sent today were just enough to get the test suite passing again (the send-email fixes) and silence any new Coverity warnings. ;)

If it does get re-rolled, I would be quite happy if my fix here just gets squashed into the appropriate commit.

-Peff

#15 lore raw

Re: leak in jt/path-filter-fix, was Re: What's cooking in git.git (Aug 2023, #01; Wed, 2)

In reply to Taylor Blau
Quoted 17 lines · Taylor Blau
Quoted 2 lines · Jeff King

Since this hit 'next', Coverity complained about a small leak. Fixed by the patch below.

The patch below looks good and makes sense to me. That warning is awfully long, though ;-).

In any event, I expect that 'next' will be rewound before this topic graduates, since it is meaty and we are in the middle of the -rc phase.

...and we also have my series on top [1], so it may be worthwhile for you, Jonathan, and I to figure out how to combine our efforts here. I think that this could likely get squashed in to Jonathan's topic if/after it gets ejected from 'next'. We can take my patches together in the same series, separately in a different one, or discard them altogether.

Quoted 8 lines

If we do decide to pursue the approach in [1], I think combining everything together into one big series makes the most sense for ease of merging.

[1]: https://lore.kernel.org/git/[email protected]/

Thanks,
Taylor

For what it's worth, whatever we decide sounds good to me.

#9 lore raw
In reply to Elijah Newren

Hi,

Quoted 19 lines · Elijah Newren
Quoted 1 line · Junio C Hamano

[...]

Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

However in a recent article
(https://github.blog/2023-07-27-scaling-merge-ort-across-github/)
GitHub says that they are already using git-replay (though not sure it's this exact version or another one), and GitLab plans to use it too.

So there are plenty folks who are capable of reviewing but they are not interested in giving it to the general public by withholding their reviews ;-)?

I can see how one could jump to that conclusion, but I don't think it's warranted. I have a little information that might shed some light:

-----

Quoted 11 lines

At the beginning of July, well before the above link came out, Johannes sent me an email pointing me at
https://github.blog/changelog/2023-06-28-rebase-commits-now-created-using-the-merge-ort-strategy/.
In the email, he also noted my earlier stated concerns on the list (about releasing replay early possibly painting us into a corner preventing some desired goals for the tool from being realized), and tried to ensure we could still work towards having a replay command like the one I envisioned while also asking for my thoughts on pushing for the current portion of work to be published and included in git. My sense was he was pushing for the work to be released, but just being careful about my concerns first.

Indeed. The patches that are running on GitHub to support rebases incorporate v3 of the git-replay patch series, and add a couple of patches on top to perform rebases similar to how things were done using libgit2 before.

Quoted 9 lines

Unfortunately, I was on vacation that week, and sadly have otherwise been buried in coming up to speed on a new team and haven't had time to even respond to git-related stuff. I didn't respond to him until late last week.

I pointed out that the 'EXPERIMENTAL' banner addresses most of my concerns so it should be fine to move forward, but I suspect my delay has resulted in him now being buried in other matters, and in the mean-time the article Christian highlighted was produced by others.

The article was produced with my input, so I cannot claim innocence.

The reason why I did not respond earlier is that I wanted to have more time to push forward with your vision of git replay, in a direction where it avoids repeating the design of git rebase.

About integrating the current patch series as-is, I would be fine with it. The best support I can offer is that this code (with minor changes that do not fundamentally modify how the core logic works) has performed gazillions of rebases in the meantime.

We just need to be mindful to keep that EXPERIMENTAL label until we fully implement the design Elijah envisions. Which might very well need the user interface to change rather drastically.

Quoted 8 lines

Anyway, hope that helps. I did review V2 and have been meaning to comment on V3 (and respond to Toon's comments), though not sure my "review" counts as such since I'm the original author of most of the patches...

(And sorry for being missing in action. I've flagged a few other emails that I'm hoping I can follow up on, but my time is currently quite limited...)

I haven't been a frequent guest on the Git mailing list, either, lately...

Ciao,
Johannes