Provide config option to expect files outside sparse patterns
PATCH 1/1 Provide config option to expect files outside sparse patterns
From: Elijah Newren <[email protected]>
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as SKIP_WORKTREE and be missing from the working tree. VFS for Git can be used to turn this expectation on its head:
all files are considered present in the working copy, though they are not vivified until first access access. With VFS for Git, most of the files do not match the sparsity patterns at first, and the VFS layer automatically updates the sparsity patterns to add more files whenever files are written.
With this background, this special usecase does not play well with the safety check we added in commit 11d46a399d ("repo_read_index: clear SKIP_WORKTREE bit from files present in worktree", 2022-01-06). Checking SKIP_WORKTREE files to see if they are present in the working tree causes them all to be immediately vivified. Further, the special VFS layer, by virtue of automatically updating the sparsity patterns and catching all accesses, isn't in need of that safety check either. Provide a configuration option, core.expectFilesOutsideSparsePatterns so that those with this special usecase can turn off the safety check.
---
Provide config option to expect files outside sparse patterns
Builds on en/present-despite-skipped, and addresses issue reported at
https://lore.kernel.org/git/[email protected]/
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1153%2Fnewren%2Fpresent-despite-skip-worktree-toggle-for-vfs-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1153/newren/present-despite-skip-worktree-toggle-for-vfs-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1153
cache.h | 1 + config.c | 5 +++++ environment.c | 1 + sparse-index.c | 3 ++- t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++ 5 files changed, 28 insertions(+), 1 deletion(-)
cache.h +1 -0
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;extern int core_apply_sparse_checkout;extern int core_sparse_checkout_cone; +extern int core_expect_files_outside_sparse_patterns;/* * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
config.c +5 -0
@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.expectfilesoutsidesparsepatterns" )) { + core_expect_files_outside_sparse_patterns =git_config_bool (var, value); +return 0 ; + } +if (!strcmp (var,"core.precomposeunicode" )) { precomposed_unicode =git_config_bool (var, value);return 0 ;
environment.c +1 -0
@@ -70,6 +70,7 @@ char *notes_ref_name;int grafts_replace_parents =1 ;int core_apply_sparse_checkout;int core_sparse_checkout_cone; +int core_expect_files_outside_sparse_patterns;int merge_log_config =-1 ;int precomposed_unicode =-1 ;/* see probe_utf8_pathname_composition() */ unsigned long pack_size_limit_cfg;
sparse-index.c +2 -1
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)int i; -if (!core_apply_sparse_checkout) +if (!core_apply_sparse_checkout || + core_expect_files_outside_sparse_patterns)return ; restart:
t/t1090-sparse-checkout-scope.sh +19 -0
@@ -84,4 +84,23 @@ test_expect_success 'in partial clone, sparse checkout only fetches needed blobs test_cmp expect actual ' +test_expect_success 'skip-worktree on files outside sparse patterns' ' + git sparse-checkout disable && + git sparse-checkout set --no-cone "a*" && + git checkout-index --all --ignore-skip-worktree-bits && + + git ls-files -t >output && + ! grep ^S output >actual && + test_must_be_empty actual && + + test_config core.expectFilesOutsideSparsePatterns true && + cat <<-\EOF >expect && + S b + S c + EOF + git ls-files -t >output && + grep ^S output >actual && + test_cmp expect actual +' + test_done
base-commit: d79d29935262f8422a6e620bd33cc2fce28e669d
Signature · 1 line
gitgitgadget
Quoted 11 lines · Elijah Newren via GitGitGadget
From: Elijah Newren <[email protected]>
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as
SKIP_WORKTREEand be missing from the working tree. VFS for Git can be used to turn this expectation on its head:
all files are considered present in the working copy, though they are not vivified until first access access. With VFS for Git, most of the files do not match the sparsity patterns at first, and the VFS layer automatically updates the sparsity patterns to add more files whenever files are written.
Quoted 9 lines
With this background, this special usecase does not play well with the safety check we added in commit
11d46a399d("repo_read_index: clearSKIP_WORKTREEbit from files present in worktree", 2022-01-06). CheckingSKIP_WORKTREEfiles to see if they are present in the working tree causes them all to be immediately vivified. Further, the special VFS layer, by virtue of automatically updating the sparsity patterns and catching all accesses, isn't in need of that safety check either. Provide a configuration option, core.expectFilesOutsideSparsePatterns so that those with this special usecase can turn off the safety check.
This patch looks like a good solution to the concerns brought up by Jonathan N. around vfsd. VFS for Git uses the microsoft/git fork with its own custom config to protect things like this. I imagine that we will start setting your core_expect_files_outside_sparse_patterns variable when reading the virtual filesystem info. We might even modify some of our custom checks to use this variable instead. That would make them appropriate to send upstream.
Should we update Documentation/config/core.txt describing this config key? Or is this intended to be an internal detail only for something like vfsd?
The only concern here really is if we want to be picky about the "VFS for Git" references instead of "vfsd" references in the commit message.
Thanks,
-Stolee
Quoted 8 lines · Derrick Stolee
This patch looks like a good solution to the concerns brought up by Jonathan N. around vfsd. VFS for Git uses the microsoft/git fork with its own custom config to protect things like this. I imagine that we will start setting your
core_expect_files_outside_sparse_patternsvariable when reading the virtual filesystem info. We might even modify some of our custom checks to use this variable instead. That would make them appropriate to send upstream.
Quoted 6 lines
Should we update
Documentation/config/core.txtdescribing this config key? Or is this intended to be an internal detail only for something like vfsd?The only concern here really is if we want to be picky about the "VFS for Git" references instead of "vfsd" references in the commit message.
Thanks for a quick response to make sure what our left hand is trying to do will not interfere with what our right hand has already been doing ;-)
Quoted 31 lines · Derrick Stolee
Quoted 11 lines · Elijah Newren via GitGitGadget
From: Elijah Newren <[email protected]>
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as
SKIP_WORKTREEand be missing from the working tree. VFS for Git can be used to turn this expectation on its head:
all files are considered present in the working copy, though they are not vivified until first access access. With VFS for Git, most of the files do not match the sparsity patterns at first, and the VFS layer automatically updates the sparsity patterns to add more files whenever files are written.Quoted 9 lines
With this background, this special usecase does not play well with the safety check we added in commit
11d46a399d("repo_read_index: clearSKIP_WORKTREEbit from files present in worktree", 2022-01-06). CheckingSKIP_WORKTREEfiles to see if they are present in the working tree causes them all to be immediately vivified. Further, the special VFS layer, by virtue of automatically updating the sparsity patterns and catching all accesses, isn't in need of that safety check either. Provide a configuration option, core.expectFilesOutsideSparsePatterns so that those with this special usecase can turn off the safety check.This patch looks like a good solution to the concerns brought up by Jonathan N. around vfsd. VFS for Git uses the microsoft/git fork with its own custom config to protect things like this. I imagine that we will start setting your
core_expect_files_outside_sparse_patternsvariable when reading the virtual filesystem info. We might even modify some of our custom checks to use this variable instead. That would make them appropriate to send upstream.
Quoted 3 lines
Should we update
Documentation/config/core.txtdescribing this config key? Or is this intended to be an internal detail only for something like vfsd?
I think it's probably only for vfsd, but vfsd users probably deserve to be able to look it up in the documentation, so it's probably worth
documenting. Once we figure out it's name and section, that is (as
per Dscho's email -- comments from you on that would be appreciated).
Quoted 2 lines
The only concern here really is if we want to be picky about the "VFS for Git" references instead of "vfsd" references in the commit message.
I'm not sure I'm understanding the distinction. Was "VFS for Git" renamed to "vfsd"? Is "vfsd" Google's modified version? Something else?
Also, vfsd doesn't seem to be Google-able whereas "VFS for Git" is, and since it's not an internal git project, it might be nice to use a name that users can find.
But anyway, I have no idea what the "correct" text here is (as probably evidenced by my questions), so if anyone wants to provide suggestions or corrections to the commit message, I'm happy to take them.
Hi Elijah,
Quoted 10 lines · Elijah Newren
Quoted 3 lines · Derrick Stolee
The only concern here really is if we want to be picky about the "VFS for Git" references instead of "vfsd" references in the commit message.
I'm not sure I'm understanding the distinction. Was "VFS for Git" renamed to "vfsd"? Is "vfsd" Google's modified version? Something else?
VFS for Git is the existing project at
https://github.com/microsoft/VFSforGit which is pretty much in maintenance mode.
vfsd was mentioned by Jonathan Tan in
https://lore.kernel.org/git/[email protected]/.
From what I gather, it is a completely separate implementation of the same ideas of VFS for Git, but from what I see it does not share any code with it (and it is unclear how much vfsd tries/tried to learn from VFS for Git, it looks like it's being done from scratch but that impression could be incorrect).
Quoted 8 lines
Also, vfsd doesn't seem to be Google-able whereas "VFS for Git" is, and since it's not an internal git project, it might be nice to use a name that users can find.
But anyway, I have no idea what the "correct" text here is (as probably evidenced by my questions), so if anyone wants to provide suggestions or corrections to the commit message, I'm happy to take them.
How about this?
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as SKIP_WORKTREE and be missing from the working tree.
There is currently an effort code-named vfsd (see
https://lore.kernel.org/git/[email protected]/
for details) to implement a Git-aware virtual file system layer (similar in spirit to the VFS for Git project) that can be used to turn this expectation on its head: all files are considered present in the working copy, though they are not vivified until actually accessed.
With such a virtual file system (VFS) layer, most of the files do not match the sparsity patterns at first, and the VFS layer automatically updates the sparsity patterns to add more files whenever files are written.
Side note: VFS for Git itself requires the Microsoft fork of Git to work, therefore this patch would only be needed in that fork, as far as VFS for Git is concerned.
Ciao,
Dscho
Quoted 19 lines · Johannes Schindelin
Hi Elijah,
Quoted 10 lines · Elijah Newren
Quoted 3 lines · Derrick Stolee
The only concern here really is if we want to be picky about the "VFS for Git" references instead of "vfsd" references in the commit message.
I'm not sure I'm understanding the distinction. Was "VFS for Git" renamed to "vfsd"? Is "vfsd" Google's modified version? Something else?
VFS for Git is the existing project at
https://github.com/microsoft/VFSforGit which is pretty much in maintenance mode.
Quoted 7 lines
vfsd was mentioned by Jonathan Tan in
https://lore.kernel.org/git/[email protected]/.
From what I gather, it is a completely separate implementation of the same ideas of VFS for Git, but from what I see it does not share any code with it (and it is unclear how much vfsd tries/tried to learn from VFS for Git, it looks like it's being done from scratch but that impression could be incorrect).
Elijah is correct that "vfsd" is not publicly visible outside of this message, so that makes it difficult for us to verify that these patches being made for it actually work as we intend.
Quoted 3 lines
Side note: VFS for Git itself requires the Microsoft fork of Git to work, therefore this patch would only be needed in that fork, as far as VFS for Git is concerned.
I would drop this side note, since a big reason for the microsoft/git fork is so we can create custom patches that handle these issues. We would probably have created a similar patch during release integration, but we are grateful to use this patch as a base for making these integrations simpler in the future.
Thanks,
-Stolee
Hi Elijah,
In addition to Stolee's feedback...
config.c static int git_default_core_config(const char *var, const char *value, void *cb) · line 1526
Quoted 12 lines · Elijah Newren via GitGitGadget
config.c +4
-0@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.expectfilesoutsidesparsepatterns" )) { + core_expect_files_outside_sparse_patterns =git_config_bool (var, value); +return 0 ; + }
The core section is already quite crowded (for which I am partially responsible, of course).
Maybe it would be a good idea to introduce the sparse section, using sparse.allowFilesMatchingPatterns or sparse.applyPatternsToWorktree = false?
Ciao,
Dscho
Quoted 22 lines · Johannes Schindelin
Hi Elijah,
In addition to Stolee's feedback...
config.c static int git_default_core_config(const char *var, const char *value, void *cb) · line 1526
Quoted 12 lines · Elijah Newren via GitGitGadget
config.c +4
-0@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.expectfilesoutsidesparsepatterns" )) { + core_expect_files_outside_sparse_patterns =git_config_bool (var, value); +return 0 ; + }The
coresection is already quite crowded (for which I am partially responsible, of course).
Quoted 3 lines
Maybe it would be a good idea to introduce the
sparsesection, usingsparse.allowFilesMatchingPatternsorsparse.applyPatternsToWorktree = false?
There's a large list of exceptions to this, but generally we have core.* for cross-command configuration, and <cmd>.* for command-specific configuration.
Excetions include http.*, mailmap.*, pack.* (arguably), trace2.*.
In this case though we have 2x /^core\.sparse[A-Z].*/ variables in git-config(1) already.
So perhaps we could name it like that, then at least you'd find all of these in the same place when looking through that documentation, although a name starting with "core.sparse" might not be otherwise ideal.
There's also no law that says we can't have a three-level variable as core.sparse.*, perhaps that would be the start of a good trend, and it would sort adjacent to core.sparse[A-Z].* ...
Quoted 29 lines · Ævar Arnfjörð Bjarmason
Quoted 22 lines · Johannes Schindelin
Hi Elijah,
In addition to Stolee's feedback...
config.c static int git_default_core_config(const char *var, const char *value, void *cb) · line 1526
Quoted 12 lines · Elijah Newren via GitGitGadget
config.c +4
-0@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.expectfilesoutsidesparsepatterns" )) { + core_expect_files_outside_sparse_patterns =git_config_bool (var, value); +return 0 ; + }The
coresection is already quite crowded (for which I am partially responsible, of course).Quoted 3 lines
Maybe it would be a good idea to introduce the
sparsesection, usingsparse.allowFilesMatchingPatternsorsparse.applyPatternsToWorktree = false?
Quoted 8 lines
There's a large list of exceptions to this, but generally we have core.* for cross-command configuration, and <cmd>.* for command-specific configuration.
Excetions include http.*, mailmap.*, pack.* (arguably), trace2.*.
In this case though we have 2x /^core\.sparse[A-Z].*/ variables in git-config(1) already.
Those wouldn't be an exception as worded. They are very much cross-command configuration variables, affecting basically every command that touches the working tree.
Quoted 8 lines
So perhaps we could name it like that, then at least you'd find all of these in the same place when looking through that documentation, although a name starting with "core.sparse" might not be otherwise ideal.
There's also no law that says we can't have a three-level variable as core.sparse.*, perhaps that would be the start of a good trend, and it would sort adjacent to core.sparse[A-Z].* ...
Interesting thought. I'm curious what others think of this.
Hi Elijah,
Quoted 8 lines · Elijah Newren
Quoted 3 lines · Ævar Arnfjörð Bjarmason
There's also no law that says we can't have a three-level variable as core.sparse.*, perhaps that would be the start of a good trend, and it would sort adjacent to core.sparse[A-Z].* ...
Interesting thought. I'm curious what others think of this.
What would make it core?
In other words, while it is valuable to consider this option, I don't think that it solves anything, for the price of additional complexity.
Let's not go there.
Ciao,
Dscho
Quoted 11 lines · Johannes Schindelin
Hi Elijah,
Quoted 7 lines · Elijah Newren
Quoted 3 lines · Ævar Arnfjörð Bjarmason
There's also no law that says we can't have a three-level variable as core.sparse.*, perhaps that would be the start of a good trend, and it would sort adjacent to core.sparse[A-Z].* ...
Quoted 6 lines
Quoted 1 line
Interesting thought. I'm curious what others think of this.
What would make it
core?In other words, while it is valuable to consider this option, I don't think that it solves anything, for the price of additional complexity.
To a first approximation what makes it core.* is that we have a core.sparseCheckout.
It's hard come up with some definition of config namespacing where it belongs there, but not a setting that tweaks it further, no?
I.e. we could quibble with the benefit of hindsight whether core.sshCommand was the best idea at the time, or if it should be remote.sshCommand or whatever.
But if we introduced a more specific sshCommandSomething having it there instead of ... well, damn, I forgot that we have ssh.variant :)
Anyway, as noted (and I feel I have to comment as the originator of this thread) I think the bikeshedding about namespacing is less important than making sure from a user perspective that these are discoverable.
E.g. core.sshCommand doesn't reference ssh.variant or the other way around, and in my "man git-config" they're at ~25% and ~90% in to what's now a *huge* document. You need to read the whole thing or have the foresight to search through it to discover both.
So for core.sshCommand and ssh.variant, and core.sparse* and sparse.* cross-linking etc. would go a long way...
Aside: I do have some local patches (waiting on a couple of other things I have outstanding) to add a "CONFIGURATION" section to all of the built-in commands with some including/splitting up of Documentation/config**.txt, so "man git-sparse-checkout" would have a section listing the subset of "git-config" applicable to the command.
Quoted 2 lines · Ævar Arnfjörð Bjarmason
To a first approximation what makes it core.* is that we have a core.sparseCheckout.
[...]
Quoted 3 lines
Anyway, as noted (and I feel I have to comment as the originator of this thread) I think the bikeshedding about namespacing is less important than making sure from a user perspective that these are discoverable.
Agreed. Though there's already also advice.updateSparsePath; even though I suspect that ctrl+F for "sparse" is what people may already be using in practice, a pointer from the git-sparse-checkout(1) page would be welcome.
Quoted 7 lines
E.g. core.sshCommand doesn't reference ssh.variant or the other way around, and in my "man git-config" they're at ~25% and ~90% in to what's now a *huge* document. You need to read the whole thing or have the foresight to search through it to discover both.
So for core.sshCommand and ssh.variant, and core.sparse* and sparse.* cross-linking etc. would go a long way...
Agreed as well.
Quoted 5 lines
Aside: I do have some local patches (waiting on a couple of other things I have outstanding) to add a "CONFIGURATION" section to all of the built-in commands with some including/splitting up of
Documentation/config**.txt, so "man git-sparse-checkout" would have a section listing the subset of "git-config" applicable to the command.
Oh! I was about to do a quick patch for git-sparse-checkout.txt, but this is even better, so I think I'll wait for you to send those. :)
Thanks,
Jonathan
Quoted 24 lines · Jonathan Nieder
Quoted 2 lines · Ævar Arnfjörð Bjarmason
To a first approximation what makes it core.* is that we have a core.sparseCheckout.
[...]
Quoted 3 lines
Anyway, as noted (and I feel I have to comment as the originator of this thread) I think the bikeshedding about namespacing is less important than making sure from a user perspective that these are discoverable.
Agreed. Though there's already also advice.updateSparsePath; even though I suspect that ctrl+F for "sparse" is what people may already be using in practice, a pointer from the git-sparse-checkout(1) page would be welcome.
Quoted 7 lines
E.g. core.sshCommand doesn't reference ssh.variant or the other way around, and in my "man git-config" they're at ~25% and ~90% in to what's now a *huge* document. You need to read the whole thing or have the foresight to search through it to discover both.
So for core.sshCommand and ssh.variant, and core.sparse* and sparse.* cross-linking etc. would go a long way...
Agreed as well.
Quoted 8 lines
Quoted 5 lines
Aside: I do have some local patches (waiting on a couple of other things I have outstanding) to add a "CONFIGURATION" section to all of the built-in commands with some including/splitting up of
Documentation/config**.txt, so "man git-sparse-checkout" would have a section listing the subset of "git-config" applicable to the command.Oh! I was about to do a quick patch for git-sparse-checkout.txt, but this is even better, so I think I'll wait for you to send those. :)
I think if you're interested in fixing just that case you shouldn't let me stop you.
The thing I have is currently dependant on [1], so a review would speed it up. I.e. needing to create new sub-dirs in Documentation/config that we lint etc. needs some of those Makefile bits (or rather, is a lot easier with them).
1. https://lore.kernel.org/git/[email protected]/
The diff for the ad-hoc series I have following that locally follows, to give you some idea.
I hacked it up before there was git-sparse*, so if I submit it I'd revisit that, and I think even for the below I stopped at some point and would need to finish it.
Documentation/Makefile | 2 + Documentation/config/color.txt | 155 ++------------------- Documentation/config/color/blame.txt | 22 +++ Documentation/config/color/branch.txt | 13 ++ Documentation/config/color/diff.txt | 27 ++++ Documentation/config/color/grep.txt | 70 ++++++++++ Documentation/config/color/log.txt | 9 ++ Documentation/config/color/push.txt | 8 ++ Documentation/config/color/showbranch.txt | 6 + Documentation/config/color/status.txt | 20 +++ Documentation/config/core.txt | 13 +- Documentation/config/core/bigFileThreshold.txt | 12 ++ Documentation/config/diff.txt | 15 -- Documentation/config/diff/difftool.txt | 14 ++ Documentation/config/difftool.txt | 7 + Documentation/config/gc.txt | 12 +- Documentation/config/gc/rerere.txt | 11 ++ Documentation/config/grep.txt | 7 +- Documentation/config/log.txt | 18 ++- Documentation/config/notes.txt | 43 +++--- Documentation/config/sendemail.txt | 40 +++++- Documentation/config/uploadpack.txt | 26 +--- Documentation/config/uploadpack/pack-objects.txt | 25 ++++ Documentation/git-add.txt | 7 + Documentation/git-am.txt | 7 + Documentation/git-apply.txt | 10 +- Documentation/git-blame.txt | 10 ++ Documentation/git-branch.txt | 6 + Documentation/git-checkout.txt | 7 + Documentation/git-clean.txt | 7 + Documentation/git-clone.txt | 9 ++ Documentation/git-commit-graph.txt | 8 ++ Documentation/git-commit.txt | 4 + Documentation/git-credential.txt | 7 + Documentation/git-diff.txt | 9 ++ Documentation/git-difftool.txt | 28 +--- Documentation/git-fast-import.txt | 7 + Documentation/git-fetch.txt | 9 ++ Documentation/git-gc.txt | 14 +- Documentation/git-grep.txt | 29 +--- Documentation/git-log.txt | 50 ++----- Documentation/git-mergetool.txt | 3 + Documentation/git-notes.txt | 54 +------ Documentation/git-pack-objects.txt | 8 +- Documentation/git-remote.txt | 15 +- Documentation/git-repack.txt | 19 +-- Documentation/git-rerere.txt | 9 ++ Documentation/git-reset.txt | 7 + Documentation/git-send-email.txt | 36 +---- Documentation/git-show-branch.txt | 7 + Documentation/githooks.txt | 4 + Documentation/includes/cmd-config-section-all.txt | 3 + Documentation/includes/cmd-config-section-rest.txt | 3 + .../templates/configuration-include-pack.txt | 10 ++ builtin/help.c | 2 +- builtin/log.c | 3 +- generate-configlist.sh | 2 +- t/t4207-log-decoration-colors.sh | 12 +- 58 files changed, 544 insertions(+), 456 deletions(-)
Documentation/Makefile +2 -0
@@ -50,7 +50,9 @@ MAN7_TXT += gitworkflows.txt HOWTO_TXT += $(wildcard howto/*.txt) DOC_DEP_TXT += $(wildcard *.txt) +DOC_DEP_TXT += $(wildcard includes/*.txt) DOC_DEP_TXT += $(wildcard config/*.txt) +DOC_DEP_TXT += $(wildcard config/*/*.txt) ifdef MAN_FILTER MAN_TXT = $(filter $(MAN_FILTER),$(MAN1_TXT) $(MAN5_TXT) $(MAN7_TXT))
Documentation/config/color.txt +13 -142
173 lines not on this page · raw message
Documentation/config/color/blame.txt new file +22 -0
@@ -0,0 +1,22 @@ +color.blame.highlightRecent:: + Specify the line annotation color for `git blame --color-by-age` + depending upon the age of the line. ++ +This setting should be set to a comma-separated list of color and +date settings, starting and ending with a color, the dates should be +set from oldest to newest. The metadata will be colored with the +specified colors if the line was introduced before the given +timestamp, overwriting older timestamped colors. ++ +Instead of an absolute timestamp relative timestamps work as well, +e.g. `2.weeks.ago` is valid to address anything older than 2 weeks. ++ +It defaults to `blue,12 month ago,white,1 month ago,red`, which +colors everything older than one year blue, recent changes between +one month and one year old are kept white, and lines introduced +within the last month are colored red. + +color.blame.repeatedLines:: + Use the specified color to colorize line annotations for + `git blame --color-lines`, if they come from the same commit as the + preceding line. Defaults to cyan.
Documentation/config/color/branch.txt new file +13 -0
@@ -0,0 +1,13 @@ +color.branch:: + A boolean to enable/disable color in the output of + linkgit:git-branch[1]. May be set to `always`, + `false` (or `never`) or `auto` (or `true`), in which case colors are used + only when the output is to a terminal. If unset, then the + value of `color.ui` is used (`auto` by default). + +color.branch.<slot>:: + Use customized color for branch coloration. `<slot>` is one of + `current` (the current branch), `local` (a local branch), + `remote` (a remote-tracking branch in refs/remotes/), + `upstream` (upstream tracking branch), `plain` (other + refs).
Documentation/config/color/diff.txt new file +27 -0
@@ -0,0 +1,27 @@ +color.diff:: + Whether to use ANSI escape sequences to add color to patches. + If this is set to `always`, linkgit:git-diff[1], + linkgit:git-log[1], and linkgit:git-show[1] will use color + for all patches. If it is set to `true` or `auto`, those + commands will only use color when output is to the terminal. + If unset, then the value of `color.ui` is used (`auto` by + default). ++ +This does not affect linkgit:git-format-patch[1] or the +'git-diff-{asterisk}' plumbing commands. Can be overridden on the +command line with the `--color[=<when>]` option. + +color.diff.<slot>:: + Use customized color for diff colorization. `<slot>` specifies + which part of the patch to use the specified color, and is one + of `context` (context text - `plain` is a historical synonym), + `meta` (metainformation), `frag` + (hunk header), 'func' (function in hunk header), `old` (removed lines), + `new` (added lines), `commit` (commit headers), `whitespace` + (highlighting whitespace errors), `oldMoved` (deleted lines), + `newMoved` (added lines), `oldMovedDimmed`, `oldMovedAlternative`, + `oldMovedAlternativeDimmed`, `newMovedDimmed`, `newMovedAlternative` + `newMovedAlternativeDimmed` (See the '<mode>' + setting of '--color-moved' in linkgit:git-diff[1] for details), + `contextDimmed`, `oldDimmed`, `newDimmed`, `contextBold`, + `oldBold`, and `newBold` (see linkgit:git-range-diff[1] for details).
Documentation/config/color/grep.txt new file +70 -0
70 lines not on this page · raw message
Documentation/config/color/log.txt new file +9 -0
@@ -0,0 +1,9 @@ +color.decorate.<slot>:: +color.logDecorate.<slot>:: + Use customized color for 'git log --decorate' output. `<slot>` is one + of `branch`, `remoteBranch`, `tag`, `stash` or `HEAD` for local + branches, remote-tracking branches, tags, stash and HEAD, respectively + and `grafted` for grafted commits. ++ +The `color.logDecorate` name is preferred to the historical +`color.decorate.
Documentation/config/color/push.txt new file +8 -0
@@ -0,0 +1,8 @@ +color.push:: + A boolean to enable/disable color in push errors. May be set to + `always`, `false` (or `never`) or `auto` (or `true`), in which + case colors are used only when the error output goes to a terminal. + If unset, then the value of `color.ui` is used (`auto` by default). + +color.push.error:: + Use customized color for push errors.
Documentation/config/color/showbranch.txt new file +6 -0
@@ -0,0 +1,6 @@ +color.showBranch:: + A boolean to enable/disable color in the output of + linkgit:git-show-branch[1]. May be set to `always`, + `false` (or `never`) or `auto` (or `true`), in which case colors are used + only when the output is to a terminal. If unset, then the + value of `color.ui` is used (`auto` by default).
Documentation/config/color/status.txt new file +20 -0
@@ -0,0 +1,20 @@ +color.status:: + A boolean to enable/disable color in the output of + linkgit:git-status[1]. May be set to `always`, + `false` (or `never`) or `auto` (or `true`), in which case colors are used + only when the output is to a terminal. If unset, then the + value of `color.ui` is used (`auto` by default). + +color.status.<slot>:: + Use customized color for status colorization. `<slot>` is + one of `header` (the header text of the status message), + `added` or `updated` (files which are added but not committed), + `changed` (files which are changed but not added in the index), + `untracked` (files which are not tracked by Git), + `branch` (the current branch), + `nobranch` (the color the 'no branch' warning is shown in, defaulting + to red), + `localBranch` or `remoteBranch` (the local and remote branch names, + respectively, when branch and tracking information is displayed in the + status short-format), or + `unmerged` (files which have unmerged changes).
Documentation/config/core.txt +1 -12
@@ -411,18 +411,7 @@ You probably do not need to adjust this value. + Common unit suffixes of 'k', 'm', or 'g' are supported. -core.bigFileThreshold:: - Files larger than this size are stored deflated, without - attempting delta compression. Storing large files without - delta compression avoids excessive memory usage, at the - slight expense of increased disk usage. Additionally files - larger than this size are always treated as binary. -+ -Default is 512 MiB on all platforms. This should be reasonable -for most projects as source code and other text files can still -be delta compressed, but larger binary media files won't be. -+ -Common unit suffixes of 'k', 'm', or 'g' are supported. +include::core/bigFileThreshold.txt[] core.excludesFile:: Specifies the pathname to the file that contains patterns to
Documentation/config/core/bigFileThreshold.txt new file +12 -0
@@ -0,0 +1,12 @@ +core.bigFileThreshold:: + Files larger than this size are stored deflated, without + attempting delta compression. Storing large files without + delta compression avoids excessive memory usage, at the + slight expense of increased disk usage. Additionally files + larger than this size are always treated as binary. ++ +Default is 512 MiB on all platforms. This should be reasonable +for most projects as source code and other text files can still +be delta compressed, but larger binary media files won't be. ++ +Common unit suffixes of 'k', 'm', or 'g' are supported.
Documentation/config/diff.txt +0 -15
@@ -178,21 +178,6 @@ diff.<driver>.cachetextconv:: Set this option to true to make the diff driver cache the text conversion outputs. See linkgit:gitattributes[5] for details. -diff.tool:: - Controls which diff tool is used by linkgit:git-difftool[1]. - This variable overrides the value configured in `merge.tool`. - The list below shows the valid built-in values. - Any other value is treated as a custom diff tool and requires - that a corresponding difftool.<tool>.cmd variable is defined. - -diff.guitool:: - Controls which diff tool is used by linkgit:git-difftool[1] when - the -g/--gui flag is specified. This variable overrides the value - configured in `merge.guitool`. The list below shows the valid - built-in values. Any other value is treated as a custom diff tool - and requires that a corresponding difftool.<guitool>.cmd variable - is defined. - include::../mergetools-diff.txt[] diff.indentHeuristic::
Documentation/config/diff/difftool.txt new file +14 -0
@@ -0,0 +1,14 @@ +diff.tool:: + Controls which diff tool is used by linkgit:git-difftool[1]. + This variable overrides the value configured in `merge.tool`. + The list below shows the valid built-in values. + Any other value is treated as a custom diff tool and requires + that a corresponding difftool.<tool>.cmd variable is defined. + +diff.guitool:: + Controls which diff tool is used by linkgit:git-difftool[1] when + the -g/--gui flag is specified. This variable overrides the value + configured in `merge.guitool`. The list below shows the valid + built-in values. Any other value is treated as a custom diff tool + and requires that a corresponding difftool.<guitool>.cmd variable + is defined.
Documentation/config/difftool.txt +7 -0
@@ -9,6 +9,13 @@ difftool.<tool>.cmd:: file containing the contents of the diff pre-image and 'REMOTE' is set to the name of the temporary file containing the contents of the diff post-image. ++ +See the `--tool=<tool>` option linkgit:git-difftool[1] for more details. difftool.prompt:: Prompt before each invocation of the diff tool. + +difftool.trustExitCode:: + Exit difftool if the invoked diff tool returns a non-zero exit status. ++ +See the `--trust-exit-code` option in linkgit:git-difftool[1] for more details.
Documentation/config/gc.txt +1 -11
@@ -123,14 +123,4 @@ or rebase occurring. Since these changes are not part of the current project most users will want to expire them sooner, which is why the default is more aggressive than `gc.reflogExpire`. -gc.rerereResolved:: - Records of conflicted merge you resolved earlier are - kept for this many days when 'git rerere gc' is run. - You can also use more human-readable "1.month.ago", etc. - The default is 60 days. See linkgit:git-rerere[1]. - -gc.rerereUnresolved:: - Records of conflicted merge you have not resolved are - kept for this many days when 'git rerere gc' is run. - You can also use more human-readable "1.month.ago", etc. - The default is 15 days. See linkgit:git-rerere[1]. +include::gc/rerere.txt[]
Documentation/config/gc/rerere.txt new file +11 -0
@@ -0,0 +1,11 @@ +gc.rerereResolved:: + Records of conflicted merge you resolved earlier are + kept for this many days when 'git rerere gc' is run. + You can also use more human-readable "1.month.ago", etc. + The default is 60 days. See linkgit:git-rerere[1]. + +gc.rerereUnresolved:: + Records of conflicted merge you have not resolved are + kept for this many days when 'git rerere gc' is run. + You can also use more human-readable "1.month.ago", etc. + The default is 15 days. See linkgit:git-rerere[1].
Documentation/config/grep.txt +5 -2
@@ -17,8 +17,11 @@ grep.extendedRegexp:: other than 'default'. grep.threads:: - Number of grep worker threads to use. - See `grep.threads` in linkgit:git-grep[1] for more information. + Number of grep worker threads to use. If unset (or set to 0), Git will + use as many threads as the number of logical cores available. + +grep.fullName:: + If set to true, enable `--full-name` option by default. grep.fallbackToNoIndex:: If set to true, fall back to git grep --no-index if git grep
Documentation/config/log.txt +13 -5
@@ -1,12 +1,11 @@ -log.abbrevCommit:: - If true, makes linkgit:git-log[1], linkgit:git-show[1], and - linkgit:git-whatchanged[1] assume `--abbrev-commit`. You may - override this option with `--no-abbrev-commit`. - log.date:: Set the default date-time mode for the 'log' command. Setting a value for log.date is similar to using 'git log''s `--date` option. See linkgit:git-log[1] for details. ++ +If the format is set to "auto:foo" and the pager is in use, format +"foo" will be the used for the date format. Otherwise "default" will +be used. log.decorate:: Print out the ref names of any commits that are shown by the log
@@ -45,6 +44,11 @@ log.showRoot:: Tools like linkgit:git-log[1] or linkgit:git-whatchanged[1], which normally hide the root commit will now show it. True by default. +log.abbrevCommit:: + If true, makes linkgit:git-log[1], linkgit:git-show[1], and + linkgit:git-whatchanged[1] assume `--abbrev-commit`. You may + override this option with `--no-abbrev-commit`. + log.showSignature:: If true, makes linkgit:git-log[1], linkgit:git-show[1], and linkgit:git-whatchanged[1] assume `--show-signature`.
@@ -53,3 +57,7 @@ log.mailmap:: If true, makes linkgit:git-log[1], linkgit:git-show[1], and linkgit:git-whatchanged[1] assume `--use-mailmap`, otherwise assume `--no-use-mailmap`. True by default. ++ +The behavior of `log.mailmap` is impacted by the setting of the more +general 'mailmap.*' configuration variables, i.e. `mailmap.file` and +`mailmap.blob`.
Documentation/config/notes.txt +26 -17
74 lines not on this page · raw message
Documentation/config/sendemail.txt +37 -3
@@ -18,17 +18,49 @@ sendemail.<identity>.*:: identity is selected, through either the command-line or `sendemail.identity`. +sendemail.multiEdit:: + If true (default), a single editor instance will be spawned to edit + files you have to edit (patches when `--annotate` is used, and the + summary when `--compose` is used). If false, files will be edited one + after the other, spawning a new editor each time. + +sendemail.confirm:: + Sets the default for whether to confirm before sending. Must be + one of 'always', 'never', 'cc', 'compose', or 'auto'. See `--confirm` + in the linkgit:git-send-email[1] documentation for the meaning of these + values. + sendemail.aliasesFile:: + To avoid typing long email addresses, point this to one or more + email aliases files. You must also supply `sendemail.aliasFileType`. + sendemail.aliasFileType:: + Format of the file(s) specified in sendemail.aliasesFile. Must be + one of 'mutt', 'mailrc', 'pine', 'elm', or 'gnus', or 'sendmail'. ++ +What an alias file in each format looks like can be found in +the documentation of the email program of the same name. The +differences and limitations from the standard formats are +described below: ++ +-- +sendmail;; +* Quoted aliases and quoted addresses are not supported: lines that + contain a `"` symbol are ignored. +* Redirection to a file (`/path/name`) or pipe (`|command`) is not + supported. +* File inclusion (`:include: /path/name`) is not supported. +* Warnings are printed on the standard error output for any + explicitly unsupported constructs, and any other lines that are not + recognized by the parser. +-- sendemail.annotate:: sendemail.bcc:: sendemail.cc:: sendemail.ccCmd:: sendemail.chainReplyTo:: -sendemail.confirm:: sendemail.envelopeSender:: sendemail.from:: -sendemail.multiEdit:: sendemail.signedoffbycc:: sendemail.smtpPass:: sendemail.suppresscc::
@@ -44,7 +76,9 @@ sendemail.thread:: sendemail.transferEncoding:: sendemail.validate:: sendemail.xmailer:: - See linkgit:git-send-email[1] for description. + These configuration variables all provide a default for + linkgit:git-send-email[1] command-line options. See its + documentation for details. sendemail.signedoffcc (deprecated):: Deprecated alias for `sendemail.signedoffbycc`.
Documentation/config/uploadpack.txt +1 -25
@@ -27,31 +27,7 @@ uploadpack.allowAnySHA1InWant:: object at all. Defaults to `false`. -uploadpack.keepAlive:: - When `upload-pack` has started `pack-objects`, there may be a - quiet period while `pack-objects` prepares the pack. Normally - it would output progress information, but if `--quiet` was used - for the fetch, `pack-objects` will output nothing at all until - the pack data begins. Some clients and networks may consider - the server to be hung and give up. Setting this option instructs - `upload-pack` to send an empty keepalive packet every - `uploadpack.keepAlive` seconds. Setting this option to 0 - disables keepalive packets entirely. The default is 5 seconds. - -uploadpack.packObjectsHook:: - If this option is set, when `upload-pack` would run - `git pack-objects` to create a packfile for a client, it will - run this shell command instead. The `pack-objects` command and - arguments it _would_ have run (including the `git pack-objects` - at the beginning) are appended to the shell command. The stdin - and stdout of the hook are treated as if `pack-objects` itself - was run. I.e., `upload-pack` will feed input intended for - `pack-objects` to the hook, and expects a completed packfile on - stdout. -+ -Note that this configuration variable is ignored if it is seen in the -repository-level config (this is a safety measure against fetching from -untrusted repositories). +include::uploadpack/pack-objects.txt[] uploadpack.allowFilter:: If this option is set, `upload-pack` will support partial
Documentation/config/uploadpack/pack-objects.txt new file +25 -0
@@ -0,0 +1,25 @@ +uploadpack.keepAlive:: + When `upload-pack` has started `pack-objects`, there may be a + quiet period while `pack-objects` prepares the pack. Normally + it would output progress information, but if `--quiet` was used + for the fetch, `pack-objects` will output nothing at all until + the pack data begins. Some clients and networks may consider + the server to be hung and give up. Setting this option instructs + `upload-pack` to send an empty keepalive packet every + `uploadpack.keepAlive` seconds. Setting this option to 0 + disables keepalive packets entirely. The default is 5 seconds. + +uploadpack.packObjectsHook:: + If this option is set, when `upload-pack` would run + `git pack-objects` to create a packfile for a client, it will + run this shell command instead. The `pack-objects` command and + arguments it _would_ have run (including the `git pack-objects` + at the beginning) are appended to the shell command. The stdin + and stdout of the hook are treated as if `pack-objects` itself + was run. I.e., `upload-pack` will feed input intended for + `pack-objects` to the hook, and expects a completed packfile on + stdout. ++ +Note that this configuration variable is ignored if it is seen in the +repository-level config (this is a safety measure against fetching from +untrusted repositories).
Documentation/git-add.txt +7 -0
@@ -431,6 +431,13 @@ they will make the patch impossible to apply: * deleting context or removal lines * modifying the contents of context or removal lines +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/add.txt[] + SEE ALSO -------- linkgit:git-status[1]
Documentation/git-am.txt +7 -0
@@ -261,6 +261,13 @@ This command can run `applypatch-msg`, `pre-applypatch`, and `post-applypatch` hooks. See linkgit:githooks[5] for more information. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/am.txt[] + SEE ALSO -------- linkgit:git-apply[1].
Documentation/git-apply.txt +3 -7
@@ -263,13 +263,9 @@ has no effect when `--index` or `--cached` is in use. CONFIGURATION ------------- -apply.ignoreWhitespace:: - Set to 'change' if you want changes in whitespace to be ignored by default. - Set to one of: no, none, never, false if you want changes in - whitespace to be significant. -apply.whitespace:: - When no `--whitespace` flag is given from the command - line, this configuration item is used as the default. +include::includes/cmd-config-section-all.txt[] + +include::config/apply.txt[] SUBMODULES ----------
Documentation/git-blame.txt +10 -0
@@ -242,6 +242,16 @@ MAPPING AUTHORS See linkgit:gitmailmap[5]. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/blame.txt[] + +include::config/color/blame.txt[] + + SEE ALSO -------- linkgit:git-annotate[1]
Documentation/git-branch.txt +6 -0
@@ -332,6 +332,12 @@ CONFIGURATION `--list` is used or implied. The default is to use a pager. See linkgit:git-config[1]. +include::includes/cmd-config-section-rest.txt[] + +include::config/branch.txt[] + +include::config/color/branch.txt[] + EXAMPLES --------
Documentation/git-checkout.txt +7 -0
@@ -600,6 +600,13 @@ $ edit frotz $ git add frotz ------------ +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/checkout.txt[] + SEE ALSO -------- linkgit:git-switch[1],
Documentation/git-clean.txt +7 -0
@@ -133,6 +133,13 @@ help:: Show brief usage of interactive git-clean. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/clean.txt[] + SEE ALSO -------- linkgit:gitignore[5]
Documentation/git-clone.txt +9 -0
@@ -358,6 +358,15 @@ $ cd my-linux $ git clone --bare -l /home/proj/.git /pub/scm/proj.git ------------ +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/init.txt[] + +include::config/clone.txt[] + GIT ---
Documentation/git-commit-graph.txt +8 -0
@@ -143,6 +143,14 @@ $ git rev-parse HEAD | git commit-graph write --stdin-commits --append ------------------------------------------------ +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/commitgraph.txt[] + + GIT --- Part of the linkgit:git[1] suite
Documentation/git-commit.txt +4 -0
@@ -557,6 +557,10 @@ The editor used to edit the commit log message will be chosen from the `VISUAL` environment variable, or the `EDITOR` environment variable (in that order). See linkgit:git-var[1] for details. +include::includes/cmd-config-section-rest.txt[] + +include::config/commit.txt[] + HOOKS ----- This command can run `commit-msg`, `prepare-commit-msg`, `pre-commit`,
Documentation/git-credential.txt +7 -0
@@ -160,6 +160,13 @@ empty string. Components which are missing from the URL (e.g., there is no username in the example above) will be left unset. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/credential.txt[] + GIT --- Part of the linkgit:git[1] suite
Documentation/git-diff.txt +9 -0
@@ -213,6 +213,15 @@ $ git diff -R <2> rewrites (very expensive). <2> Output diff in reverse. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/diff.txt[] + +include::config/color/diff.txt[] + SEE ALSO -------- diff(1),
Documentation/git-difftool.txt +6 -22
@@ -113,33 +113,17 @@ instead. `--no-symlinks` is the default on Windows. See linkgit:git-diff[1] for the full list of supported options. -CONFIG VARIABLES ----------------- +CONFIGURATION +------------- + 'git difftool' falls back to 'git mergetool' config variables when the difftool equivalents have not been defined. -diff.tool:: - The default diff tool to use. - -diff.guitool:: - The default diff tool to use when `--gui` is specified. +include::includes/cmd-config-section-rest.txt[] -difftool.<tool>.path:: - Override the path for the given tool. This is useful in case - your tool is not in the PATH. - -difftool.<tool>.cmd:: - Specify the command to invoke the specified diff tool. -+ -See the `--tool=<tool>` option above for more details. +include::config/diff/difftool.txt[] -difftool.prompt:: - Prompt before each invocation of the diff tool. - -difftool.trustExitCode:: - Exit difftool if the invoked diff tool returns a non-zero exit status. -+ -See the `--trust-exit-code` option above for more details. +include::config/difftool.txt[] SEE ALSO --------
Documentation/git-fast-import.txt +7 -0
@@ -1564,6 +1564,13 @@ operator can use this facility to peek at the objects and refs from an import in progress, at the cost of some added running time and worse compression. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/fastimport.txt[] + SEE ALSO -------- linkgit:git-fast-export[1]
Documentation/git-fetch.txt +9 -0
@@ -285,6 +285,15 @@ linkgit:git-gc[1]). include::transfer-data-leaks.txt[] +CONFIGURATION +------------- + +include::includes/cmd-config-section-rest.txt[] + +include::config/fetch.txt[] + +include::config/remote.txt[] + BUGS ---- Using --recurse-submodules can only fetch new commits in already checked
Documentation/git-gc.txt +6 -8
@@ -103,14 +103,6 @@ more time, and the resulting space/delta optimization may or may not be worth it. Not using this at all is the right trade-off for most users and their repositories. -CONFIGURATION -------------- - -The below documentation is the same as what's found in -linkgit:git-config[1]: - -include::config/gc.txt[] - NOTES -----
@@ -150,6 +142,12 @@ HOOKS The 'git gc --auto' command will run the 'pre-auto-gc' hook. See linkgit:githooks[5] for more information. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/gc.txt[] SEE ALSO --------
Documentation/git-grep.txt +2 -27
@@ -334,34 +334,9 @@ performance in this case, it might be desirable to use `--threads=1`. CONFIGURATION ------------- -grep.lineNumber:: - If set to true, enable `-n` option by default. - -grep.column:: - If set to true, enable the `--column` option by default. - -grep.patternType:: - Set the default matching behavior. Using a value of 'basic', 'extended', - 'fixed', or 'perl' will enable the `--basic-regexp`, `--extended-regexp`, - `--fixed-strings`, or `--perl-regexp` option accordingly, while the - value 'default' will return to the default matching behavior. - -grep.extendedRegexp:: - If set to true, enable `--extended-regexp` option by default. This - option is ignored when the `grep.patternType` option is set to a value - other than 'default'. - -grep.threads:: - Number of grep worker threads to use. If unset (or set to 0), Git will - use as many threads as the number of logical cores available. - -grep.fullName:: - If set to true, enable `--full-name` option by default. - -grep.fallbackToNoIndex:: - If set to true, fall back to git grep --no-index if git grep - is executed outside of a git repository. Defaults to false. +include::includes/cmd-config-section-all.txt[] +include::config/grep.txt[] GIT ---
Documentation/git-log.txt +9 -41
62 lines not on this page · raw message
Documentation/git-mergetool.txt +3 -0
@@ -101,6 +101,9 @@ success of the resolution after the custom tool has exited. CONFIGURATION ------------- + +include::includes/cmd-config-section-all.txt[] + include::config/mergetool.txt[] TEMPORARY FILES
Documentation/git-notes.txt +3 -51
67 lines not on this page · raw message
Documentation/git-pack-objects.txt +3 -5
@@ -414,12 +414,10 @@ over user-wide config, and so forth). CONFIGURATION ------------- -Various configuration variables affect packing, see -linkgit:git-config[1] (search for "pack" and "delta"). +include::templates/configuration-include-pack.txt[] + +include::config/uploadpack/pack-objects.txt[] -Notably, delta compression is not used on objects larger than the -`core.bigFileThreshold` configuration variable and on files with the -attribute `delta` set to false. SEE ALSO --------
Documentation/git-remote.txt +8 -7
@@ -196,13 +196,6 @@ be updated. (See linkgit:git-config[1]). With `--prune` option, run pruning against all the remotes that are updated. -DISCUSSION ----------- - -The remote configuration is achieved using the `remote.origin.url` and -`remote.origin.fetch` configuration variables. (See -linkgit:git-config[1]). - EXIT STATUS -----------
@@ -255,6 +248,14 @@ $ git remote add -f -t master -m master origin git://example.com/git.git/ $ git merge origin ------------ +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/remote.txt[] + +include::config/remotes.txt[] SEE ALSO --------
Documentation/git-repack.txt +3 -16
@@ -204,22 +204,9 @@ linkgit:git-multi-pack-index[1]). CONFIGURATION ------------- -Various configuration variables affect packing, see -linkgit:git-config[1] (search for "pack" and "delta"). - -By default, the command passes `--delta-base-offset` option to -'git pack-objects'; this typically results in slightly smaller packs, -but the generated packs are incompatible with versions of Git older than -version 1.4.4. If you need to share your repository with such ancient Git -versions, either directly or via the dumb http protocol, then you -need to set the configuration variable `repack.UseDeltaBaseOffset` to -"false" and repack. Access from old Git versions over the native protocol -is unaffected by this option as the conversion is performed on the fly -as needed in that case. - -Delta compression is not used on objects larger than the -`core.bigFileThreshold` configuration variable and on files with the -attribute `delta` set to false. +include::templates/configuration-include-pack.txt[] + +include::config/repack.txt[] SEE ALSO --------
Documentation/git-rerere.txt +9 -0
@@ -217,6 +217,15 @@ same as lines with conflict markers, 'git rerere' may fail to record a conflict resolution. To work around this, the `conflict-marker-size` setting in linkgit:gitattributes[5] can be used. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/rerere.txt[] + +include::config/gc.txt[] + GIT --- Part of the linkgit:git[1] suite
Documentation/git-reset.txt +7 -0
@@ -499,6 +499,13 @@ working index HEAD target working index HEAD `X` means any state and `U` means an unmerged index. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/reset.txt[] + GIT --- Part of the linkgit:git[1] suite
Documentation/git-send-email.txt +2 -34
@@ -456,41 +456,9 @@ Information CONFIGURATION ------------- -sendemail.aliasesFile:: - To avoid typing long email addresses, point this to one or more - email aliases files. You must also supply `sendemail.aliasFileType`. +include::includes/cmd-config-section-all.txt[] -sendemail.aliasFileType:: - Format of the file(s) specified in sendemail.aliasesFile. Must be - one of 'mutt', 'mailrc', 'pine', 'elm', or 'gnus', or 'sendmail'. -+ -What an alias file in each format looks like can be found in -the documentation of the email program of the same name. The -differences and limitations from the standard formats are -described below: -+ --- -sendmail;; -* Quoted aliases and quoted addresses are not supported: lines that - contain a `"` symbol are ignored. -* Redirection to a file (`/path/name`) or pipe (`|command`) is not - supported. -* File inclusion (`:include: /path/name`) is not supported. -* Warnings are printed on the standard error output for any - explicitly unsupported constructs, and any other lines that are not - recognized by the parser. --- - -sendemail.multiEdit:: - If true (default), a single editor instance will be spawned to edit - files you have to edit (patches when `--annotate` is used, and the - summary when `--compose` is used). If false, files will be edited one - after the other, spawning a new editor each time. - -sendemail.confirm:: - Sets the default for whether to confirm before sending. Must be - one of 'always', 'never', 'cc', 'compose', or 'auto'. See `--confirm` - in the previous section for the meaning of these values. +include::config/sendemail.txt[] EXAMPLES --------
Documentation/git-show-branch.txt +7 -0
@@ -199,6 +199,13 @@ shows 10 reflog entries going back from the tip as of 1 hour ago. Without `--list`, the output also shows how these tips are topologically related with each other. +CONFIGURATION +------------- + +include::includes/cmd-config-section-all.txt[] + +include::config/showbranch.txt[] + GIT --- Part of the linkgit:git[1] suite
Documentation/githooks.txt +4 -0
@@ -702,6 +702,10 @@ SEE ALSO -------- linkgit:git-hook[1] +The `uploadpack.packObjectsHook` configuration variable can configure +a hook for linkgit:git-pack-objects[1] to use, but not in a way that +uses the hook facility discussed above. + GIT --- Part of the linkgit:git[1] suite
Documentation/includes/cmd-config-section-all.txt new file +3 -0
@@ -0,0 +1,3 @@ +Everything below this line in this section is selectively included +from the linkgit:git-config[1] documentation. The content is the same +as what's found there:
Documentation/includes/cmd-config-section-rest.txt new file +3 -0
@@ -0,0 +1,3 @@ +The discussion above this line is unique to this documentation, but +everything below is included from section(s) in linkgit:git-config[1] +aplicable to this command:
Documentation/templates/configuration-include-pack.txt new file +10 -0
@@ -0,0 +1,10 @@ +Various configuration variables affect packing. Those variables and +more are discussed in linkgit:git-config[1]. + +The below documentation is selectively included from +linkgit:git-config[1]. It should be all the configuration variables +relevant to this command, but some may have been missed. When in doubt +doubt consult linkgit:git-config[1] for the full listing: + +include::../config/core/bigFileThreshold.txt[] +include::../config/pack.txt[]
builtin/help.c +1 -1
@@ -94,7 +94,7 @@ static void list_config_help(enum show_config_type type)struct slot_expansion slot_expansions[] = { {"advice" ,"*" , list_config_advices }, {"color.branch" ,"<slot>" , list_config_color_branch_slots }, - {"color.decorate" ,"<slot>" , list_config_color_decorate_slots }, + {"color.logDecorate" ,"<slot>" , list_config_color_decorate_slots }, {"color.diff" ,"<slot>" , list_config_color_diff_slots }, {"color.grep" ,"<slot>" , list_config_color_grep_slots }, {"color.interactive" ,"<slot>" , list_config_color_interactive_slots },
builtin/log.c +2 -1
@@ -522,7 +522,8 @@ static int git_log_config(const char *var, const char *value, void *cb) default_follow =git_config_bool (var, value);return 0 ; } -if (skip_prefix (var,"color.decorate." , &slot_name)) +if (skip_prefix (var,"color.logDecorate." , &slot_name) || +skip_prefix (var,"color.decorate." , &slot_name))return parse_decorate_color_config (var, slot_name, value);if (!strcmp (var,"log.mailmap" )) { use_mailmap_config =git_config_bool (var, value);
generate-configlist.sh +1 -1
@@ -7,7 +7,7 @@ print_config_list () { cat <<EOF static const char *config_name_list[] = { EOF - grep -h '^[a-zA-Z].*\..*::$' Documentation/*config.txt Documentation/config/*.txt | + grep -h '^[a-zA-Z].*\..*::$' Documentation/*config.txt Documentation/config/*.txt Documentation/config/*/*.txt| sed '/deprecated/d; s/::$//; s/, */\n/g' | sort | sed 's/^.*$/ "&",/'
t/t4207-log-decoration-colors.sh +9 -3
@@ -56,10 +56,16 @@ EOF # We want log to show all, but the second parent to refs/stash is irrelevant # to this test since it does not contain any decoration, hence --first-parent test_expect_success 'Commit Decorations Colored Correctly' ' - git log --first-parent --abbrev=10 --all --decorate --oneline --color=always | - sed "s/[0-9a-f]\{10,10\}/COMMIT_ID/" | - test_decode_color >out && + git log --first-parent --abbrev=10 --all --decorate --oneline --color=always >out.raw && + sed "s/[0-9a-f]\{10,10\}/COMMIT_ID/" <out.raw >out.color && + test_decode_color <out.color >out && test_cmp expected out ' +test_expect_success 'color.logDecorate.* is preferred to color.decorate.*' ' + git config --rename-section color.decorate color.logDecorate && + git log --first-parent --abbrev=10 --all --decorate --oneline --color=always >out.raw2 && + test_cmp out.raw out.raw2 +' + test_done
Quoted 23 lines · Johannes Schindelin
Hi Elijah,
In addition to Stolee's feedback...
config.c static int git_default_core_config(const char *var, const char *value, void *cb) · line 1526
Quoted 12 lines · Elijah Newren via GitGitGadget
config.c +4
-0@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.expectfilesoutsidesparsepatterns" )) { + core_expect_files_outside_sparse_patterns =git_config_bool (var, value); +return 0 ; + }The
coresection is already quite crowded (for which I am partially responsible, of course).
Quoted 3 lines
Maybe it would be a good idea to introduce the
sparsesection, usingsparse.allowFilesMatchingPatternsorsparse.applyPatternsToWorktree = false?
That's a fair point. At one point Stolee wanted to change from core.sparse* to sparse.* -- but by that point we already had users and would have had to deal with a bit of a migration story (and wondering what to do if people had both old and new config variables set inconsistently).
I'm not sure if it's optimal to try to keep the sparse settings together (thus put new ones under core), or try to avoid filling core. I guess if we moved towards sparse.* now, it might be an easier migration story if we only have two options to move. And besides, we're already split between multiple sections with extensions.worktreeConfig, core.sparseCheckout{,Cone}, and index.sparse already...so maybe adding one more section would be par for the course. ;-)
So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd like to hear Stolee's thoughts too.
Quoted 44 lines · Elijah Newren
Quoted 23 lines · Johannes Schindelin
Hi Elijah,
In addition to Stolee's feedback...
config.c static int git_default_core_config(const char *var, const char *value, void *cb) · line 1526
Quoted 12 lines · Elijah Newren via GitGitGadget
config.c +4
-0@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.expectfilesoutsidesparsepatterns" )) { + core_expect_files_outside_sparse_patterns =git_config_bool (var, value); +return 0 ; + }The
coresection is already quite crowded (for which I am partially responsible, of course).Quoted 3 lines
Maybe it would be a good idea to introduce the
sparsesection, usingsparse.allowFilesMatchingPatternsorsparse.applyPatternsToWorktree = false?That's a fair point. At one point Stolee wanted to change from core.sparse* to sparse.* -- but by that point we already had users and would have had to deal with a bit of a migration story (and wondering what to do if people had both old and new config variables set inconsistently).
I'm not sure if it's optimal to try to keep the sparse settings together (thus put new ones under core), or try to avoid filling core. I guess if we moved towards sparse.* now, it might be an easier migration story if we only have two options to move. And besides, we're already split between multiple sections with extensions.worktreeConfig, core.sparseCheckout{,Cone}, and index.sparse already...so maybe adding one more section would be par for the course. ;-)
Quoted 2 lines
So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd like to hear Stolee's thoughts too.
Yes, maybe sparse.* etc. is best. I'm fine with whatever you two come up with.
I just wanted to raise the point that it's less important to consider names/namespaces for variables one-by-one, and more important that all the variables in a "family" fit well together, the users can find them all in one place etc. (ideally, not always possible).
Hi Elijah,
Quoted 27 lines · Elijah Newren
Quoted 21 lines · Johannes Schindelin
In addition to Stolee's feedback...
config.c static int git_default_core_config(const char *var, const char *value, void *cb) · line 1526
Quoted 12 lines · Elijah Newren via GitGitGadget
config.c +4
-0@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.expectfilesoutsidesparsepatterns" )) { + core_expect_files_outside_sparse_patterns =git_config_bool (var, value); +return 0 ; + }The
coresection is already quite crowded (for which I am partially responsible, of course).Quoted 3 lines
Maybe it would be a good idea to introduce the
sparsesection, usingsparse.allowFilesMatchingPatternsorsparse.applyPatternsToWorktree = false?
Quoted 5 lines
That's a fair point. At one point Stolee wanted to change from core.sparse* to sparse.* -- but by that point we already had users and would have had to deal with a bit of a migration story (and wondering what to do if people had both old and new config variables set inconsistently).
Right, migration is always hard.
And it's outside of the scope of this here patch series, of course.
Quoted 8 lines
I'm not sure if it's optimal to try to keep the sparse settings together (thus put new ones under core), or try to avoid filling core. I guess if we moved towards sparse.* now, it might be an easier migration story if we only have two options to move. And besides, we're already split between multiple sections with extensions.worktreeConfig, core.sparseCheckout{,Cone}, and index.sparse already...so maybe adding one more section would be par for the course. ;-)
FWIW as a potential #leftoverbits, we could migrate those to sparse.* where sparse.* would take precendence over core.sparse* and the usual deprecation notice would be shown via the advice mechanism.
Quoted 2 lines
So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd like to hear Stolee's thoughts too.
Indeed, his opinion weighs more than mine on this matter.
Ciao,
Dscho
Quoted 44 lines · Elijah Newren
Quoted 23 lines · Johannes Schindelin
Hi Elijah,
In addition to Stolee's feedback...
config.c static int git_default_core_config(const char *var, const char *value, void *cb) · line 1526
Quoted 12 lines · Elijah Newren via GitGitGadget
config.c +4
-0@@ -1520,6 +1520,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return 0 ; } +if (!strcmp (var,"core.expectfilesoutsidesparsepatterns" )) { + core_expect_files_outside_sparse_patterns =git_config_bool (var, value); +return 0 ; + }The
coresection is already quite crowded (for which I am partially responsible, of course).Quoted 3 lines
Maybe it would be a good idea to introduce the
sparsesection, usingsparse.allowFilesMatchingPatternsorsparse.applyPatternsToWorktree = false?That's a fair point. At one point Stolee wanted to change from core.sparse* to sparse.* -- but by that point we already had users and would have had to deal with a bit of a migration story (and wondering what to do if people had both old and new config variables set inconsistently).
I'm not sure if it's optimal to try to keep the sparse settings together (thus put new ones under core), or try to avoid filling core. I guess if we moved towards sparse.* now, it might be an easier migration story if we only have two options to move. And besides, we're already split between multiple sections with extensions.worktreeConfig, core.sparseCheckout{,Cone}, and index.sparse already...so maybe adding one more section would be par for the course. ;-)
Quoted 2 lines
So, I'm leaning towards sparse.expectFilesOutsideOfPatterns, but I'd like to hear Stolee's thoughts too.
This has been an interesting discussion.
I think that the existence of core.sparseCheckout[Cone] isn't a good reason to keep adding to the core.* namespace. Creating a sparse.* namespace is a good idea. I doubt this will be the last time we want a config option for some behavior custom to sparse-checkouts (or virtual environments).
I agree that migrating core.sparseCheckout[Cone] to the sparse.* namespace would need to be done carefully. In particular, we would probably need to _always_ understand the core.* versions, but start to prefer the sparse.* versions if there is a conflict. This work could be delayed indefinitely, or could be offered as a project for a new contributor (if we experienced contributors could agree on a clear path to make this transition).
Thanks,
-Stolee
PATCH v2 1/1 repo_read_index: add config to expect files outside sparse patterns
In reply to Elijah Newren via GitGitGadgetFrom: Elijah Newren <[email protected]>
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as SKIP_WORKTREE and be missing from the working tree. In edge cases, this can be violated and cause confusion, so in a sparse checkout, since 11d46a399d ("repo_read_index: clear SKIP_WORKTREE bit from files present in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE bit at read time for entries corresponding to files that are present in the working tree.
However, there is a more atypical situation where this situation would be expected. A Git-aware virtual file system[1] takes advantage of its position as a file system driver to expose all files in the working tree, fetch them on demand using partial clone on access, and tell Git to pay attention to them on demand by updating the sparse checkout pattern on writes. This means that commands like "git status" only has to examine files that have potentially been modified, whereas commands like "ls" are able to show the entire codebase without requiring manual updates to the sparse checkout pattern.
Thus since 11d46a399d, Git with such Git-aware virtual file systems unsets the SKIP_WORKTREE bit for all files and commands like "git status" have to fetch and examine them all.
Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to allow limiting the tracked set of files to a small set once again. A Git-aware virtual file system or other application that wants to maintain files outside of the sparse checkout can set this in a repository to instruct Git not to check for the presence of SKIP_WORKTREE files. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from interrupted updates that forget to delete some files or unset SKIP_WORKTREE for them.
[1] such as the vfsd described in
https://lore.kernel.org/git/[email protected]/
[jn: fleshed out commit message and documentation, added missing
include to config.txt, moved to a separate config callback]
---
Some minor updates, but this is basically the same as the patch you sent. Thoughts?
Thanks,
Jonathan
Documentation/config.txt | 2 ++ Documentation/config/sparse.txt | 24 ++++++++++++++++++++++++ cache.h | 1 + config.c | 14 ++++++++++++++ environment.c | 1 + sparse-index.c | 3 ++- t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++ 7 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/sparse.txt
Documentation/config.txt +2 -0
@@ -468,6 +468,8 @@ include::config/sequencer.txt[] include::config/showbranch.txt[] +include::config/sparse.txt[] + include::config/splitindex.txt[] include::config/ssh.txt[]
Documentation/config/sparse.txt new file +24 -0
@@ -0,0 +1,24 @@ +sparse.expectFilesOutsideOfPatterns:: + Typically with sparse checkouts, files not matching any + sparsity patterns are marked as such in the index file and + missing from the working tree. Accordingly, Git will + ordinarily check whether files that the index indicates are + outside of the sparse area are present in the working tree and + mark them as present in the index if so. This option can be + used to tell Git that such present-but-unmatching files are + expected and to stop checking for them. ++ +The default is `false`. Leaving this set to `false` is recommended in +most situations because it allows Git to recover from an interrupted +operation that updated the working tree without updating the index or +vice versa. ++ +A Git-based virtual file system (VFS) can turn the usual expectation +on its head: files are present in the working copy but do not take +up much disk space because their contents are not downloaded until +they are accessed. With such a virtual file system layer, most files +do not match the sparsity patterns at first, and the VFS layer +updates the sparsity patterns to add more files whenever files are +written. Setting this to `true` supports such a setup where files are +expected to be present outside the sparse area and a separate, robust +mechanism is responsible for keeping the sparsity patterns up to date.
cache.h +1 -0
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;extern int core_apply_sparse_checkout;extern int core_sparse_checkout_cone; +extern int sparse_expect_files_outside_of_patterns;/* * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
config.c +14 -0
@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return platform_core_config (var, value, cb); } +static int git_default_sparse_config (const char *var,const char *value) +{ +if (!strcmp (var,"sparse.expectfilesoutsideofpatterns" )) { + sparse_expect_files_outside_of_patterns =git_config_bool (var, value); +return 0 ; + } + +/* Add other config variables here and to Documentation/config/sparse.txt. */ +return 0 ; +} +static int git_default_i18n_config (const char *var,const char *value) {if (!strcmp (var,"i18n.commitencoding" ))
@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)return 0 ; } +if (starts_with (var,"sparse." )) +return git_default_sparse_config (var, value); +/* Add other config variables here and to Documentation/config.txt. */ return 0 ; }
environment.c +1 -0
@@ -70,6 +70,7 @@ char *notes_ref_name;int grafts_replace_parents =1 ;int core_apply_sparse_checkout;int core_sparse_checkout_cone; +int sparse_expect_files_outside_of_patterns;int merge_log_config =-1 ;int precomposed_unicode =-1 ;/* see probe_utf8_pathname_composition() */ unsigned long pack_size_limit_cfg;
sparse-index.c +2 -1
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)int i; -if (!core_apply_sparse_checkout) +if (!core_apply_sparse_checkout || + sparse_expect_files_outside_of_patterns)return ; restart:
t/t1090-sparse-checkout-scope.sh +19 -0
@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' ' test "$(cat b)" = "modified" ' +test_expect_success 'skip-worktree on files outside sparse patterns' ' + git sparse-checkout disable && + git sparse-checkout set --no-cone "a*" && + git checkout-index --all --ignore-skip-worktree-bits && + + git ls-files -t >output && + ! grep ^S output >actual && + test_must_be_empty actual && + + test_config sparse.expectFilesOutsideOfPatterns true && + cat <<-\EOF >expect && + S b + S c + EOF + git ls-files -t >output && + grep ^S output >actual && + test_cmp expect actual +' + test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' ' test_create_repo server && git clone "file://$(pwd)/server" client &&
Signature · 1 line
2.35.1.574.g5d30c73bfb
Re: [PATCH v2] repo_read_index: add config to expect files outside sparse patterns
Quoted 11 lines · Jonathan Nieder
From: Elijah Newren <[email protected]>
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as
SKIP_WORKTREEand be missing from the working tree. In edge cases, this can be violated and cause confusion, so in a sparse checkout, since11d46a399d("repo_read_index: clearSKIP_WORKTREEbit from files present in worktree", 2022-01-06), Git automatically clears theSKIP_WORKTREEbit at read time for entries corresponding to files that are present in the working tree.
Quoted 9 lines
However, there is a more atypical situation where this situation would be expected. A Git-aware virtual file system[1] takes advantage of its position as a file system driver to expose all files in the working tree, fetch them on demand using partial clone on access, and tell Git to pay attention to them on demand by updating the sparse checkout pattern on writes. This means that commands like "git status" only has to examine files that have potentially been modified, whereas commands like "ls" are able to show the entire codebase without requiring manual updates to the sparse checkout pattern.
Should that be s/commands/a command/ or else s/only has/only have/?
Quoted 4 lines
Thus since
11d46a399d, Git with such Git-aware virtual file systems unsets theSKIP_WORKTREEbit for all files and commands like "git status" have to fetch and examine them all.
Quoted 8 lines
Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to allow limiting the tracked set of files to a small set once again. A Git-aware virtual file system or other application that wants to maintain files outside of the sparse checkout can set this in a repository to instruct Git not to check for the presence of
SKIP_WORKTREEfiles. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from interrupted updates that forget to
Please don't presume that these only come from interrupted updates. As per the referenced 11d46a399d that started all this:
"""
There are various ways for users to get files to be present in the working copy despite having the SKIP_WORKTREE bit set for that file in the index. This may come from:
* various git commands not really supporting the SKIP_WORKTREE bit[1,2]
* users grabbing files from elsewhere and writing them to the worktree (perhaps even cached in their editor)
* users attempting to "abort" a sparse-checkout operation with a not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the working tree is not atomic)[3].
"""
Quoted 1 line
delete some files or unset
SKIP_WORKTREEfor them.
The only problem 11d46a399d corrects is having the SKIP_WORKTREE being *set* despite the file being present. So the "recover from...updates that...unset SKIP_WORKTREE" doesn't make any sense.
Quoted 11 lines
[1] such as the vfsd described in
https://lore.kernel.org/git/[email protected]/[jn: fleshed out commit message and documentation, added missing
include to config.txt, moved to a separate config callback]Signed-off-by Elijah Newren <[email protected]>Signed-off-by Jonathan Nieder <[email protected]>---
Some minor updates, but this is basically the same as the patch you sent. Thoughts?
...and it looks even more like the v2 I was about to send, so it seems like we're pretty much on the same page. :-)
Overall, it looks good, and your version has some things that are nicer than mine, but I did have a couple small notes on the commit message above and a similar one on the config description below.
Quoted 36 lines
Thanks,
JonathanDocumentation/config.txt | 2 ++ Documentation/config/sparse.txt | 24 ++++++++++++++++++++++++ cache.h | 1 + config.c | 14 ++++++++++++++ environment.c | 1 + sparse-index.c | 3 ++- t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++ 7 files changed, 63 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/sparse.txtDocumentation/config.txt +2
-0@@ -468,6 +468,8 @@ include::config/sequencer.txt[] include::config/showbranch.txt[] +include::config/sparse.txt[] + include::config/splitindex.txt[] include::config/ssh.txt[]Documentation/config/sparse.txt new file +4
-0@@ -0,0 +1,24 @@ +sparse.expectFilesOutsideOfPatterns:: + Typically with sparse checkouts, files not matching any + sparsity patterns are marked as such in the index file and + missing from the working tree. Accordingly, Git will
Documentation/config/sparse.txt line 14
Quoted 10 lines
Documentation/config/sparse.txt new file +10
-0+ ordinarily check whether files that the index indicates are + outside of the sparse area are present in the working tree and + mark them as present in the index if so. This option can be + used to tell Git that such present-but-unmatching files are + expected and to stop checking for them. ++ +The default is `false`. Leaving this set to `false` is recommended in +most situations because it allows Git to recover from an interrupted +operation that updated the working tree without updating the index or +vice versa.
Again, please don't claim this is only for recovering from interrupted operations; there are other cases -- Git commands that write the working tree but not the index (checkout-index, git apply), and users mucking around with files, for example.
Quoted 112 lines
Documentation/config/sparse.txt new file +10
-0++ +A Git-based virtual file system (VFS) can turn the usual expectation +on its head: files are present in the working copy but do not take +up much disk space because their contents are not downloaded until +they are accessed. With such a virtual file system layer, most files +do not match the sparsity patterns at first, and the VFS layer +updates the sparsity patterns to add more files whenever files are +written. Setting this to `true` supports such a setup where files are +expected to be present outside the sparse area and a separate, robust +mechanism is responsible for keeping the sparsity patterns up to date.cache.h +1
-0@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;extern int core_apply_sparse_checkout;extern int core_sparse_checkout_cone; +extern int sparse_expect_files_outside_of_patterns;/* * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value). config.c +14
-0@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return platform_core_config (var, value, cb); } +static int git_default_sparse_config (const char *var,const char *value) +{ +if (!strcmp (var,"sparse.expectfilesoutsideofpatterns" )) { + sparse_expect_files_outside_of_patterns =git_config_bool (var, value); +return 0 ; + } + +/* Add other config variables here and to Documentation/config/sparse.txt. */ +return 0 ; +} +static int git_default_i18n_config (const char *var,const char *value) {if (!strcmp (var,"i18n.commitencoding" ))@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)return 0 ; } +if (starts_with (var,"sparse." )) +return git_default_sparse_config (var, value); +/* Add other config variables here and to Documentation/config.txt. */ return 0 ; }environment.c +1
-0@@ -70,6 +70,7 @@ char *notes_ref_name;int grafts_replace_parents =1 ;int core_apply_sparse_checkout;int core_sparse_checkout_cone; +int sparse_expect_files_outside_of_patterns;int merge_log_config =-1 ;int precomposed_unicode =-1 ;/* see probe_utf8_pathname_composition() */ unsigned long pack_size_limit_cfg;sparse-index.c +2
-1@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)int i; -if (!core_apply_sparse_checkout) +if (!core_apply_sparse_checkout || + sparse_expect_files_outside_of_patterns)return ; restart:t/t1090-sparse-checkout-scope.sh +19
-1@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' ' test "$(cat b)" = "modified" ' +test_expect_success 'skip-worktree on files outside sparse patterns' ' + git sparse-checkout disable && + git sparse-checkout set --no-cone "a*" && + git checkout-index --all --ignore-skip-worktree-bits && + + git ls-files -t >output && + ! grep ^S output >actual && + test_must_be_empty actual && + + test_config sparse.expectFilesOutsideOfPatterns true && + cat <<-\EOF >expect && + S b + S c + EOF + git ls-files -t >output && + grep ^S output >actual && + test_cmp expect actual +' + test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' ' test_create_repo server && git clone "file://$(pwd)/server" client && --2.35.1.574.g5d30c73bfb
PATCH v3 1/1 repo_read_index: add config to expect files outside sparse patterns
In reply to Jonathan NiederTypically with sparse checkouts, we expect files outside the sparsity patterns to be marked as SKIP_WORKTREE and be missing from the working tree. In edge cases, this can be violated and cause confusion, so in a sparse checkout, since 11d46a399d ("repo_read_index: clear SKIP_WORKTREE bit from files present in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE bit at read time for entries corresponding to files that are present in the working tree.
However, there is a more atypical situation where this situation would be expected. A Git-aware virtual file system[1] takes advantage of its position as a file system driver to expose all files in the working tree, fetch them on demand using partial clone on access, and tell Git to pay attention to them on demand by updating the sparse checkout pattern on writes. This means that commands like "git status" only have to examine files that have potentially been modified, whereas commands like "ls" are able to show the entire codebase without requiring manual updates to the sparse checkout pattern.
Thus since 11d46a399d, Git with such Git-aware virtual file systems unsets the SKIP_WORKTREE bit for all files and commands like "git status" have to fetch and examine them all.
Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to allow limiting the tracked set of files to a small set once again. A Git-aware virtual file system or other application that wants to maintain files outside of the sparse checkout can set this in a repository to instruct Git not to check for the presence of SKIP_WORKTREE files. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from the variety of difficult issues detailed in 11d46a399d for paths with SKIP_WORKTREE set despite the path being present.
[1] such as the vfsd described in
https://lore.kernel.org/git/[email protected]/
---
Changes since v2:
* Made tweaks to the commit message and the text of the config option as highlighted in my response to Jonathan's v2.
I'm guessing that since there are no code (only documentation) changes since Jonathan's v2 submission, that this patch satisfies vfsd/Google's needs. I'm also guessing it matches what Stolee and Dscho stated in their comments on v1. But it'd be nice to have an ack from each side just to make sure.
Documentation/config.txt | 2 ++ Documentation/config/sparse.txt | 28 ++++++++++++++++++++++++++++ cache.h | 1 + config.c | 14 ++++++++++++++ environment.c | 1 + sparse-index.c | 3 ++- t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++ 7 files changed, 67 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/sparse.txt
Documentation/config.txt +2 -0
@@ -468,6 +468,8 @@ include::config/sequencer.txt[] include::config/showbranch.txt[] +include::config/sparse.txt[] + include::config/splitindex.txt[] include::config/ssh.txt[]
Documentation/config/sparse.txt new file +28 -0
@@ -0,0 +1,28 @@ +sparse.expectFilesOutsideOfPatterns:: + Typically with sparse checkouts, files not matching any + sparsity patterns are marked as such in the index file and + missing from the working tree. Accordingly, Git will + ordinarily check whether files that the index indicates are + outside of the sparse area are present in the working tree and + mark them as present in the index if so. This option can be + used to tell Git that such present-but-unmatching files are + expected and to stop checking for them. ++ +The default is `false`. Paths which are marked as SKIP_WORKTREE +despite being present (which can occur for a few different reasons) +typically present a range of problems which are difficult for users to +discover and recover from. The default setting avoids such issues. ++ +A Git-based virtual file system (VFS) can turn the usual expectation +on its head: files are present in the working copy but do not take +up much disk space because their contents are not downloaded until +they are accessed. With such a virtual file system layer, most files +do not match the sparsity patterns at first, and the VFS layer +updates the sparsity patterns to add more files whenever files are +written. Setting this to `true` supports such a setup where files are +expected to be present outside the sparse area and a separate, robust +mechanism is responsible for keeping the sparsity patterns up to date. ++ +Note that the checking and clearing of the SKIP_WORKTREE bit only +happens when core.sparseCheckout is true, so this config option has no +effect unless core.sparseCheckout is true.
cache.h +1 -0
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;extern int core_apply_sparse_checkout;extern int core_sparse_checkout_cone; +extern int sparse_expect_files_outside_of_patterns;/* * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
config.c +14 -0
@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return platform_core_config (var, value, cb); } +static int git_default_sparse_config (const char *var,const char *value) +{ +if (!strcmp (var,"sparse.expectfilesoutsideofpatterns" )) { + sparse_expect_files_outside_of_patterns =git_config_bool (var, value); +return 0 ; + } + +/* Add other config variables here and to Documentation/config/sparse.txt. */ +return 0 ; +} +static int git_default_i18n_config (const char *var,const char *value) {if (!strcmp (var,"i18n.commitencoding" ))
@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)return 0 ; } +if (starts_with (var,"sparse." )) +return git_default_sparse_config (var, value); +/* Add other config variables here and to Documentation/config.txt. */ return 0 ; }
environment.c +1 -0
@@ -70,6 +70,7 @@ char *notes_ref_name;int grafts_replace_parents =1 ;int core_apply_sparse_checkout;int core_sparse_checkout_cone; +int sparse_expect_files_outside_of_patterns;int merge_log_config =-1 ;int precomposed_unicode =-1 ;/* see probe_utf8_pathname_composition() */ unsigned long pack_size_limit_cfg;
sparse-index.c +2 -1
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)int i; -if (!core_apply_sparse_checkout) +if (!core_apply_sparse_checkout || + sparse_expect_files_outside_of_patterns)return ; restart:
t/t1090-sparse-checkout-scope.sh +19 -0
@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' ' test "$(cat b)" = "modified" ' +test_expect_success 'skip-worktree on files outside sparse patterns' ' + git sparse-checkout disable && + git sparse-checkout set --no-cone "a*" && + git checkout-index --all --ignore-skip-worktree-bits && + + git ls-files -t >output && + ! grep ^S output >actual && + test_must_be_empty actual && + + test_config sparse.expectFilesOutsideOfPatterns true && + cat <<-\EOF >expect && + S b + S c + EOF + git ls-files -t >output && + grep ^S output >actual && + test_cmp expect actual +' + test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' ' test_create_repo server && git clone "file://$(pwd)/server" client &&
Signature · 1 line
2.34.0.22.g587c05af5b.dirty
Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns
Quoted 4 lines · Elijah Newren
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as
SKIP_WORKTREEand be missing from the working tree. In edge cases, this can be violated and cause confusion, so in a sparse checkout, since11d46a399d("repo_read_index: clearSKIP_WORKTREE
I think this refers to af6a5187 (repo_read_index: clear SKIP_WORKTREE bit from files present in worktree, 2022-01-14).
Quoted 3 lines
bit from files present in worktree", 2022-01-06), Git automatically clears the
SKIP_WORKTREEbit at read time for entries corresponding to files that are present in the working tree.
So, this is a workflow where the user deliberately "creates" these files outside the sparsity cone or pattern (by various non-automated means like editing, copying/renaming, or untarring). If they did so on purpose, they may be interested in comparing them with existing commits, or even including them as a newer version in the next commit they create. To help that workflow, clearing the bit makes sense.
Am I on the right path? I am wondering if mentioning some of that would help understanding by the reader when it is contrasted with the (competing) goal of supporting VFS use case mentioned next.
Quoted 1 line
However, there is a more atypical situation where this situation would
I wonder if that is "more atypical" (read: makes me wonder if it depends on who the reader is what is typoical), and more importantly, if it helps understanding of the reader (read: whether which one is more common, we'd want to support both camps anyway).
There is another workflow, however, that it is expected that paths outside the sparsity patterns appear to exist in the working tree and that they do not lose the SKIP_WORKTREE bit, at least until they get modified.
or something?
Quoted 8 lines
be expected. A Git-aware virtual file system[1] takes advantage of its position as a file system driver to expose all files in the working tree, fetch them on demand using partial clone on access, and tell Git to pay attention to them on demand by updating the sparse checkout pattern on writes. This means that commands like "git status" only have to examine files that have potentially been modified, whereas commands like "ls" are able to show the entire codebase without requiring manual updates to the sparse checkout pattern.
Well explained.
Quoted 1 line
Thus since
11d46a399d, Git with such Git-aware virtual file systems
The same stale reference.
Quoted 12 lines
unsets the
SKIP_WORKTREEbit for all files and commands like "git status" have to fetch and examine them all.Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to allow limiting the tracked set of files to a small set once again. A Git-aware virtual file system or other application that wants to maintain files outside of the sparse checkout can set this in a repository to instruct Git not to check for the presence of
SKIP_WORKTREEfiles. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from the variety of difficult issues detailed in11d46a399dfor paths withSKIP_WORKTREEset despite the path being
Ditto.
Quoted 4 lines
I'm guessing that since there are no code (only documentation) changes since Jonathan's v2 submission, that this patch satisfies vfsd/Google's needs. I'm also guessing it matches what Stolee and Dscho stated in their comments on v1. But it'd be nice to have an ack from each side just to make sure.
True. Let me queue but leave it just outside 'next' until that happens.
I think the name of the knob is what Jonathan suggested, so I presume that their side would be fine with it, but I am curious (I do not wonder, though) what the plan on the Microsoft's side going forward. When they update the version of Git bundled in their vfsd, would this be reverted and an equivalent they have (and they may have more such "workaround" in other areas as well?) will be kept, so whatever we do here will add a minor inconvenience to them but will not hurt them otherwise?
Documentation/config/sparse.txt line 3
Quoted 9 lines
Documentation/config/sparse.txt new file +3
-0@@ -0,0 +1,28 @@ +sparse.expectFilesOutsideOfPatterns:: + Typically with sparse checkouts, files not matching any + sparsity patterns are marked as such in the index file and
s/index file/index/ perhaps.
Documentation/config/sparse.txt line 7
Quoted 4 lines
Documentation/config/sparse.txt new file +4
-0+ missing from the working tree. Accordingly, Git will + ordinarily check whether files that the index indicates are + outside of the sparse area are present in the working tree and + mark them as present in the index if so. This option can be
Just an observation. According to this sentence, "sparse area" is "paths that ought to be present in the working tree", so paths "outside of the sparse area" that are present need to be corrected to be "in" the sparse area by futzing bits. I always get confused when I hear "sparse area" if the author meant "paths that ought to be missing" or "present", but maybe it is just me.
Documentation/config/sparse.txt line 9
Quoted 2 lines
Documentation/config/sparse.txt new file +2
-0+ used to tell Git that such present-but-unmatching files are + expected and to stop checking for them.
OK.
Documentation/config/sparse.txt line 13
Quoted 4 lines
Documentation/config/sparse.txt new file +4
-0++ +The default is `false`. Paths which are marked as SKIP_WORKTREE +despite being present (which can occur for a few different reasons) +typically present a range of problems which are difficult for users to
s/typically // perhaps.
Documentation/config/sparse.txt line 24
Quoted 11 lines
Documentation/config/sparse.txt new file +11
-0+discover and recover from. The default setting avoids such issues. ++ +A Git-based virtual file system (VFS) can turn the usual expectation +on its head: files are present in the working copy but do not take +up much disk space because their contents are not downloaded until +they are accessed. With such a virtual file system layer, most files +do not match the sparsity patterns at first, and the VFS layer +updates the sparsity patterns to add more files whenever files are +written. Setting this to `true` supports such a setup where files are +expected to be present outside the sparse area and a separate, robust +mechanism is responsible for keeping the sparsity patterns up to date.
s/separate, robust/separate/ I would think.
We make the outside mechanism that makes these files appear to be present to also be responsible for maintaining the sparse bit and patterns.
When the user (or IDE) sets this knob to 'true', do we even have to expect that files appear to be present? In the use case we intend to support with this feature, i.e. some VFS, we might expect all paths to appear to be present, but if that VFS also allows users to configure to expose only a subset of paths, not all paths may appear to be present. And we are perfectly OK with that, becuase we do not expect anything about the working tree paths outside the sparsity pattern. Am I mistaken?
So, "... supports such a setup where some external system releaves us of the responsibility of maintaining the consistency between the presence of working tree files and sparsity patterns, so we stop expecting whether files are present or missing outside the sparse area", might be closer to the truth?
Documentation/config/sparse.txt line 28
Quoted 3 lines
Documentation/config/sparse.txt new file +3
-0+Note that the checking and clearing of the SKIP_WORKTREE bit only +happens when core.sparseCheckout is true, so this config option has no +effect unless core.sparseCheckout is true.
Good note to have. There is no mention of "cone" mode in the entire description; it is unclear if this only applies to "pattern" mode or to both "pattern" and "cone" modes, which may want to be clarified.
Thanks.
Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns
Quoted 10 lines · Junio C Hamano
Quoted 4 lines · Elijah Newren
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as
SKIP_WORKTREEand be missing from the working tree. In edge cases, this can be violated and cause confusion, so in a sparse checkout, since11d46a399d("repo_read_index: clearSKIP_WORKTREEI think this refers to
af6a5187(repo_read_index: clearSKIP_WORKTREEbit from files present in worktree, 2022-01-14).
Yes, I'm usually pretty good about grabbing the commits you created and merged rather than the local copies I submitted, but I messed it up here. Thanks for catching.
Quoted 13 lines
Quoted 4 lines
bit from files present in worktree", 2022-01-06), Git automatically clears the
SKIP_WORKTREEbit at read time for entries corresponding to files that are present in the working tree.So, this is a workflow where the user deliberately "creates" these files outside the sparsity cone or pattern (by various non-automated means like editing, copying/renaming, or untarring). If they did so on purpose, they may be interested in comparing them with existing commits, or even including them as a newer version in the next commit they create. To help that workflow, clearing the bit makes sense.
Quoted 3 lines
Am I on the right path? I am wondering if mentioning some of that would help understanding by the reader when it is contrasted with the (competing) goal of supporting VFS use case mentioned next.
Yes, this is one of three ways that things can get out of sync. Since this commit was being added to en/present-despite-skipped which spelled this out in detail and thus would appear just a few commits before, I thought it wasn't worth repeating these details, but I guessed wrong. I'll include them here again.
Quoted 7 lines
Quoted 1 line
However, there is a more atypical situation where this situation would
I wonder if that is "more atypical" (read: makes me wonder if it depends on who the reader is what is typoical), and more importantly, if it helps understanding of the reader (read: whether which one is more common, we'd want to support both camps anyway).
Quoted 6 lines
There is another workflow, however, that it is expected that paths outside the sparsity patterns appear to exist in the working tree and that they do not lose the
SKIP_WORKTREEbit, at least until they get modified.or something?
I like it.
Quoted 28 lines
Quoted 8 lines
be expected. A Git-aware virtual file system[1] takes advantage of its position as a file system driver to expose all files in the working tree, fetch them on demand using partial clone on access, and tell Git to pay attention to them on demand by updating the sparse checkout pattern on writes. This means that commands like "git status" only have to examine files that have potentially been modified, whereas commands like "ls" are able to show the entire codebase without requiring manual updates to the sparse checkout pattern.
Well explained.
Quoted 1 line
Thus since
11d46a399d, Git with such Git-aware virtual file systemsThe same stale reference.
Quoted 12 lines
unsets the
SKIP_WORKTREEbit for all files and commands like "git status" have to fetch and examine them all.Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to allow limiting the tracked set of files to a small set once again. A Git-aware virtual file system or other application that wants to maintain files outside of the sparse checkout can set this in a repository to instruct Git not to check for the presence of
SKIP_WORKTREEfiles. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from the variety of difficult issues detailed in11d46a399dfor paths withSKIP_WORKTREEset despite the path being
Quoted 1 line
Ditto.
Will fix all three.
Quoted 17 lines
Quoted 4 lines
I'm guessing that since there are no code (only documentation) changes since Jonathan's v2 submission, that this patch satisfies vfsd/Google's needs. I'm also guessing it matches what Stolee and Dscho stated in their comments on v1. But it'd be nice to have an ack from each side just to make sure.
True. Let me queue but leave it just outside 'next' until that happens.
I think the name of the knob is what Jonathan suggested, so I presume that their side would be fine with it, but I am curious (I do not wonder, though) what the plan on the Microsoft's side going forward. When they update the version of Git bundled in their vfsd, would this be reverted and an equivalent they have (and they may have more such "workaround" in other areas as well?) will be kept, so whatever we do here will add a minor inconvenience to them but will not hurt them otherwise?
Quoted 11 lines
Documentation/config/sparse.txt line 3
Quoted 9 lines
Documentation/config/sparse.txt new file +3
-0@@ -0,0 +1,28 @@ +sparse.expectFilesOutsideOfPatterns:: + Typically with sparse checkouts, files not matching any + sparsity patterns are marked as such in the index file ands/index file/index/ perhaps.
Will fix.
Quoted 11 lines
Documentation/config/sparse.txt line 7
Quoted 4 lines
Documentation/config/sparse.txt new file +4
-0+ missing from the working tree. Accordingly, Git will + ordinarily check whether files that the index indicates are + outside of the sparse area are present in the working tree and + mark them as present in the index if so. This option can beJust an observation. According to this sentence, "sparse area" is "paths that ought to be present in the working tree", so paths "outside of the sparse area" that are present need to be corrected to be "in" the sparse area by futzing bits. I always get confused when I hear "sparse area" if the author meant "paths that ought to be missing" or "present", but maybe it is just me.
I reworded this based on a combination of the feedback from you and Jonathan. I think it's clearer now; I'll resubmit soon.
Quoted 4 lines
Documentation/config/sparse.txt line 9
Quoted 3 lines
Documentation/config/sparse.txt new file +2
-0+ used to tell Git that such present-but-unmatching files are + expected and to stop checking for them.
Quoted 8 lines
OK.
Documentation/config/sparse.txt line 13
Quoted 4 lines
Documentation/config/sparse.txt new file +4
-0++ +The default is `false`. Paths which are marked as SKIP_WORKTREE +despite being present (which can occur for a few different reasons) +typically present a range of problems which are difficult for users tos/typically // perhaps.
Sure.
Quoted 27 lines
Documentation/config/sparse.txt line 24
Quoted 11 lines
Documentation/config/sparse.txt new file +11
-0+discover and recover from. The default setting avoids such issues. ++ +A Git-based virtual file system (VFS) can turn the usual expectation +on its head: files are present in the working copy but do not take +up much disk space because their contents are not downloaded until +they are accessed. With such a virtual file system layer, most files +do not match the sparsity patterns at first, and the VFS layer +updates the sparsity patterns to add more files whenever files are +written. Setting this to `true` supports such a setup where files are +expected to be present outside the sparse area and a separate, robust +mechanism is responsible for keeping the sparsity patterns up to date.s/separate, robust/separate/ I would think.
We make the outside mechanism that makes these files appear to be present to also be responsible for maintaining the sparse bit and patterns.
When the user (or IDE) sets this knob to 'true', do we even have to expect that files appear to be present? In the use case we intend to support with this feature, i.e. some VFS, we might expect all paths to appear to be present, but if that VFS also allows users to configure to expose only a subset of paths, not all paths may appear to be present. And we are perfectly OK with that, becuase we do not expect anything about the working tree paths outside the sparsity pattern. Am I mistaken?
Quoted 5 lines
So, "... supports such a setup where some external system releaves us of the responsibility of maintaining the consistency between the presence of working tree files and sparsity patterns, so we stop expecting whether files are present or missing outside the sparse area", might be closer to the truth?
Good point, and thanks for the suggested wording.
Quoted 7 lines
Documentation/config/sparse.txt line 28
Quoted 3 lines
Documentation/config/sparse.txt new file +3
-0+Note that the checking and clearing of the SKIP_WORKTREE bit only +happens when core.sparseCheckout is true, so this config option has no +effect unless core.sparseCheckout is true.Good note to have. There is no mention of "cone" mode in the entire description; it is unclear if this only applies to "pattern" mode or to both "pattern" and "cone" modes, which may want to be clarified.
Yeah, it applies to both pattern and cone modes. I went with Jonathan's wording, which I think sounded more precise and suggested that only core.sparseCheckout=true matters:
"""
Regardless of this setting, Git does not check for
present-but-unmatching files unless sparse checkout is enabled, so this config option has no effect unless core.sparseCheckout is true.
"""
Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns
In reply to Elijah NewrenHi,
Quoted 1 line · Elijah Newren
Signed-off-by Elijah Newren <[email protected]>
Thanks, and sorry for the slow review. My one remaining area for nits is the documentation, but that can be improved iteratively via patches on top.
[...]
Documentation/config/sparse.txt line 6
Quoted 9 lines
Documentation/config/sparse.txt new file +6
-0@@ -0,0 +1,28 @@ +sparse.expectFilesOutsideOfPatterns:: + Typically with sparse checkouts, files not matching any + sparsity patterns are marked as such in the index file and + missing from the working tree. Accordingly, Git will + ordinarily check whether files that the index indicates are + outside of the sparse area are present in the working tree and
Junio mentioned the "sparse area" could suggest that the area is itself sparse and devoid of files, so it might not have been the best choice of words on my part. Perhaps "whether files that the index indicates are not checked out are present in the working tree" would work here?
Documentation/config/sparse.txt line 14
Quoted 8 lines
Documentation/config/sparse.txt new file +8
-0+ mark them as present in the index if so. This option can be + used to tell Git that such present-but-unmatching files are + expected and to stop checking for them. ++ +The default is `false`. Paths which are marked as SKIP_WORKTREE +despite being present (which can occur for a few different reasons) +typically present a range of problems which are difficult for users to +discover and recover from. The default setting avoids such issues.
The git-sparse-checkout(1) page never describes what SKIP_WORKTREE means, so it might not be obvious to them what this means. Also, the "can occur for a few different reasons" may leave the user wondering whether they are subject to those reasons. What the reader wants to know is "I should keep using the default because it makes Git work better", so how about something like
The default is false, which allows Git to automatically recover from the list of files in the index and working tree falling out of sync.
+
?
Documentation/config/sparse.txt line 24
Quoted 10 lines
Documentation/config/sparse.txt new file +10
-0++ +A Git-based virtual file system (VFS) can turn the usual expectation +on its head: files are present in the working copy but do not take +up much disk space because their contents are not downloaded until +they are accessed. With such a virtual file system layer, most files +do not match the sparsity patterns at first, and the VFS layer +updates the sparsity patterns to add more files whenever files are +written. Setting this to `true` supports such a setup where files are +expected to be present outside the sparse area and a separate, robust +mechanism is responsible for keeping the sparsity patterns up to date.
Here I spent most of the words explaining what a Git-based VFS layer is, which is also not too relevant to most users (who are just interested in "is true the right value for me?"). How about reducing it to the following?
Set this to true if you are in a setup where extra files are expected to be present and a separate, robust mechanism is responsible for keeping the sparsity patterns up to date, such as a Git-aware virtual file system.
?
Documentation/config/sparse.txt line 28
Quoted 4 lines
Documentation/config/sparse.txt new file +4
-0++ +Note that the checking and clearing of the SKIP_WORKTREE bit only +happens when core.sparseCheckout is true, so this config option has no +effect unless core.sparseCheckout is true.
Good note. Same nit about the user not necessarily knowing what SKIP_WORKTREE means applies. Also, we can remove the extra words "Note that" since the dutiful reader should be noting everything we say. :) I think that would make
+
Regardless of this setting, Git does not check for present-but-unmatching files unless sparse checkout is enabled, so this config option has no effect unless core.sparseCheckout is true.
Thanks,
Jonathan
Re: [PATCH v3] repo_read_index: add config to expect files outside sparse patterns
Quoted 14 lines · Jonathan Nieder
Hi,
Quoted 1 line · Elijah Newren
Signed-off-by Elijah Newren <[email protected]>Reviewed-by Jonathan Nieder <[email protected]>Thanks, and sorry for the slow review. My one remaining area for nits is the documentation, but that can be improved iteratively via patches on top.
[...]
Quoted 15 lines
Documentation/config/sparse.txt line 6
Quoted 9 lines
Documentation/config/sparse.txt new file +6
-0@@ -0,0 +1,28 @@ +sparse.expectFilesOutsideOfPatterns:: + Typically with sparse checkouts, files not matching any + sparsity patterns are marked as such in the index file and + missing from the working tree. Accordingly, Git will + ordinarily check whether files that the index indicates are + outside of the sparse area are present in the working tree andJunio mentioned the "sparse area" could suggest that the area is itself sparse and devoid of files, so it might not have been the best choice of words on my part. Perhaps "whether files that the index indicates are not checked out are present in the working tree" would work here?
I rewrote the paragraph. I think it's more clear now; I'll resubmit it here soon.
Quoted 16 lines
Documentation/config/sparse.txt line 14
Quoted 8 lines
Documentation/config/sparse.txt new file +8
-0+ mark them as present in the index if so. This option can be + used to tell Git that such present-but-unmatching files are + expected and to stop checking for them. ++ +The default is `false`. Paths which are marked as SKIP_WORKTREE +despite being present (which can occur for a few different reasons) +typically present a range of problems which are difficult for users to +discover and recover from. The default setting avoids such issues.The git-sparse-checkout(1) page never describes what
SKIP_WORKTREEmeans, so it might not be obvious to them what this means. Also, the "can occur for a few different reasons" may leave the user wondering whether they are subject to those reasons. What the reader wants to know is "I should keep using the default because it makes Git work better", so how about something like
Quoted 6 lines
The default is
false, which allows Git to automatically recover from the list of files in the index and working tree falling out of sync.
+?
I like this.
Quoted 16 lines
Documentation/config/sparse.txt line 24
Quoted 10 lines
Documentation/config/sparse.txt new file +10
-0++ +A Git-based virtual file system (VFS) can turn the usual expectation +on its head: files are present in the working copy but do not take +up much disk space because their contents are not downloaded until +they are accessed. With such a virtual file system layer, most files +do not match the sparsity patterns at first, and the VFS layer +updates the sparsity patterns to add more files whenever files are +written. Setting this to `true` supports such a setup where files are +expected to be present outside the sparse area and a separate, robust +mechanism is responsible for keeping the sparsity patterns up to date.Here I spent most of the words explaining what a Git-based VFS layer is, which is also not too relevant to most users (who are just interested in "is
truethe right value for me?"). How about reducing it to the following?
Quoted 6 lines
Set this to
trueif you are in a setup where extra files are expected to be present and a separate, robust mechanism is responsible for keeping the sparsity patterns up to date, such as a Git-aware virtual file system.?
I like this, but I also added in some of the wording suggestions from Junio here, so it's
a bit longer but has both some of his suggested wording and yours for slightly different aspects that I think works well together.
Quoted 11 lines
Documentation/config/sparse.txt line 28
Quoted 5 lines
Documentation/config/sparse.txt new file +4
-0++ +Note that the checking and clearing of the SKIP_WORKTREE bit only +happens when core.sparseCheckout is true, so this config option has no +effect unless core.sparseCheckout is true.Good note. Same nit about the user not necessarily knowing what
SKIP_WORKTREEmeans applies. Also, we can remove the extra words "Note that" since the dutiful reader should be noting everything we say. :) I think that would make
Quoted 5 lines
+
Regardless of this setting, Git does not check for present-but-unmatching files unless sparse checkout is enabled, so this config option has no effect unlesscore.sparseCheckoutistrue.
I like this too. Thanks for the suggestions, the proposed changes, and the review.
PATCH v4 1/1 repo_read_index: add config to expect files outside sparse patterns
In reply to Elijah NewrenTypically with sparse checkouts, we expect files outside the sparsity patterns to be marked as SKIP_WORKTREE and be missing from the working tree. Sometimes this expectation would be violated however; including in cases such as:
* users grabbing files from elsewhere and writing them to the worktree (perhaps by editing a cached copy in an editor, copying/renaming, or even untarring)
* various git commands having incomplete or no support for the
SKIP_WORKTREE bit[1,2]
* users attempting to "abort" a sparse-checkout operation with a not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the working tree is not atomic)[3].
When the SKIP_WORKTREE bit in the index did not reflect the presence of the file in the working tree, it traditionally caused confusion and was difficult to detect and recover from. So, in a sparse checkout, since af6a51875a ("repo_read_index: clear SKIP_WORKTREE bit from files present in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE bit at index read time for entries corresponding to files that are present in the working tree.
There is another workflow, however, where it is expected that paths outside the sparsity patterns appear to exist in the working tree and that they do not lose the SKIP_WORKTREE bit, at least until they get modified. A Git-aware virtual file system[4] takes advantage of its position as a file system driver to expose all files in the working tree, fetch them on demand using partial clone on access, and tell Git to pay attention to them on demand by updating the sparse checkout pattern on writes. This means that commands like "git status" only have to examine files that have potentially been modified, whereas commands like "ls" are able to show the entire codebase without requiring manual updates to the sparse checkout pattern.
Thus since af6a51875a, Git with such Git-aware virtual file systems unsets the SKIP_WORKTREE bit for all files and commands like "git status" have to fetch and examine them all.
Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to allow limiting the tracked set of files to a small set once again. A Git-aware virtual file system or other application that wants to maintain files outside of the sparse checkout can set this in a repository to instruct Git not to check for the presence of SKIP_WORKTREE files. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from the variety of difficult issues detailed in af6a51875a for paths with SKIP_WORKTREE set despite the path being present.
[1] https://lore.kernel.org/git/[email protected]/
[2] The three long paragraphs in the middle of
https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
[1] such as the vfsd described in
https://lore.kernel.org/git/[email protected]/
---
Changes since v3:
* Many wording updates based on feedback and suggestions from both Jonathan and Junio. Thanks for providing concrete suggestions!
Range-diff:
1: 92a13a8f08 ! 1: 5ded1532cd repo_read_index: add config to expect files outside sparse patterns
@@ Commit message
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as SKIP_WORKTREE and be missing from the working
- tree. In edge cases, this can be violated and cause confusion, so in a
- sparse checkout, since 11d46a399d ("repo_read_index: clear SKIP_WORKTREE
- bit from files present in worktree", 2022-01-06), Git automatically
- clears the SKIP_WORKTREE bit at read time for entries corresponding to
- files that are present in the working tree.
+ tree. Sometimes this expectation would be violated however; including
+ in cases such as:
+ * users grabbing files from elsewhere and writing them to the worktree
+ (perhaps by editing a cached copy in an editor, copying/renaming, or
+ even untarring)
+ * various git commands having incomplete or no support for the
+ SKIP_WORKTREE bit[1,2]
+ * users attempting to "abort" a sparse-checkout operation with a
+ not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the
+ working tree is not atomic)[3].
+ When the SKIP_WORKTREE bit in the index did not reflect the presence of
+ the file in the working tree, it traditionally caused confusion and was
+ difficult to detect and recover from. So, in a sparse checkout, since
+ af6a51875a ("repo_read_index: clear SKIP_WORKTREE bit from files present
+ in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE
+ bit at index read time for entries corresponding to files that are
+ present in the working tree.
- However, there is a more atypical situation where this situation would
- be expected. A Git-aware virtual file system[1] takes advantage of its
+ There is another workflow, however, where it is expected that paths
+ outside the sparsity patterns appear to exist in the working tree and
+ that they do not lose the SKIP_WORKTREE bit, at least until they get
+ modified. A Git-aware virtual file system[4] takes advantage of its
position as a file system driver to expose all files in the working tree, fetch them on demand using partial clone on access, and tell Git to pay attention to them on demand by updating the sparse checkout
@@ Commit message
like "ls" are able to show the entire codebase without requiring manual updates to the sparse checkout pattern.
- Thus since 11d46a399d, Git with such Git-aware virtual file systems
+ Thus since af6a51875a, Git with such Git-aware virtual file systems
unsets the SKIP_WORKTREE bit for all files and commands like "git status" have to fetch and examine them all.
@@ Commit message
SKIP_WORKTREE files. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from the variety of difficult issues detailed in
- 11d46a399d for paths with SKIP_WORKTREE set despite the path being
+ af6a51875a for paths with SKIP_WORKTREE set despite the path being
present.
+ [1] https://lore.kernel.org/git/[email protected]/
+ [2] The three long paragraphs in the middle of
+ https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
+ [3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
[1] such as the vfsd described in
https://lore.kernel.org/git/[email protected]/
- Helped-by: Jonathan Nieder <[email protected]>
+ Reviewed-by: Jonathan Nieder <[email protected]>
## Documentation/config.txt ##
@@ Documentation/config.txt: include::config/sequencer.txt[]
@@ Documentation/config/sparse.txt (new)
@@
+sparse.expectFilesOutsideOfPatterns::
+ Typically with sparse checkouts, files not matching any
-+ sparsity patterns are marked as such in the index file and
-+ missing from the working tree. Accordingly, Git will
-+ ordinarily check whether files that the index indicates are
-+ outside of the sparse area are present in the working tree and
-+ mark them as present in the index if so. This option can be
-+ used to tell Git that such present-but-unmatching files are
-+ expected and to stop checking for them.
++ sparsity patterns are marked with a SKIP_WORKTREE bit in the
++ index and are missing from the working tree. Accordingly, Git
++ will ordinarily check whether files with the SKIP_WORKTREE bit
++ are in fact present in the working tree contrary to
++ expectations. If Git finds any, it marks those paths as
++ present by clearing the relevant SKIP_WORKTREE bits. This
++ option can be used to tell Git that such
++ present-despite-skipped files are expected and to stop
++ checking for them.
++
-+The default is false. Paths which are marked as SKIP_WORKTREE -+despite being present (which can occur for a few different reasons) -+typically present a range of problems which are difficult for users to -+discover and recover from. The default setting avoids such issues. ++The default is false, which allows Git to automatically recover ++from the list of files in the index and working tree falling out of ++sync.
++
-+A Git-based virtual file system (VFS) can turn the usual expectation -+on its head: files are present in the working copy but do not take -+up much disk space because their contents are not downloaded until -+they are accessed. With such a virtual file system layer, most files -+do not match the sparsity patterns at first, and the VFS layer -+updates the sparsity patterns to add more files whenever files are -+written. Setting this to true supports such a setup where files are -+expected to be present outside the sparse area and a separate, robust -+mechanism is responsible for keeping the sparsity patterns up to date. ++Set this to true if you are in a setup where some external factor ++relieves Git of the responsibility for maintaining the consistency ++between the presence of working tree files and sparsity patterns. For ++example, if you have a Git-aware virtual file system that has a robust ++mechanism for keeping the working tree and the sparsity patterns up to ++date based on access patterns.
++
-+Note that the checking and clearing of the SKIP_WORKTREE bit only -+happens when core.sparseCheckout is true, so this config option has no -+effect unless core.sparseCheckout is true.
++Regardless of this setting, Git does not check for ++present-despite-skipped files unless sparse checkout is enabled, so ++this config option has no effect unless core.sparseCheckout is ++true.
## cache.h ##
@@ cache.h: extern const char *core_fsmonitor;
Documentation/config.txt | 2 ++ Documentation/config/sparse.txt | 27 +++++++++++++++++++++++++++ cache.h | 1 + config.c | 14 ++++++++++++++ environment.c | 1 + sparse-index.c | 3 ++- t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++ 7 files changed, 66 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/sparse.txt
Documentation/config.txt +2 -0
@@ -468,6 +468,8 @@ include::config/sequencer.txt[] include::config/showbranch.txt[] +include::config/sparse.txt[] + include::config/splitindex.txt[] include::config/ssh.txt[]
Documentation/config/sparse.txt new file +27 -0
@@ -0,0 +1,27 @@ +sparse.expectFilesOutsideOfPatterns:: + Typically with sparse checkouts, files not matching any + sparsity patterns are marked with a SKIP_WORKTREE bit in the + index and are missing from the working tree. Accordingly, Git + will ordinarily check whether files with the SKIP_WORKTREE bit + are in fact present in the working tree contrary to + expectations. If Git finds any, it marks those paths as + present by clearing the relevant SKIP_WORKTREE bits. This + option can be used to tell Git that such + present-despite-skipped files are expected and to stop + checking for them. ++ +The default is `false`, which allows Git to automatically recover +from the list of files in the index and working tree falling out of +sync. ++ +Set this to `true` if you are in a setup where some external factor +relieves Git of the responsibility for maintaining the consistency +between the presence of working tree files and sparsity patterns. For +example, if you have a Git-aware virtual file system that has a robust +mechanism for keeping the working tree and the sparsity patterns up to +date based on access patterns. ++ +Regardless of this setting, Git does not check for +present-despite-skipped files unless sparse checkout is enabled, so +this config option has no effect unless `core.sparseCheckout` is +`true`.
cache.h +1 -0
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;extern int core_apply_sparse_checkout;extern int core_sparse_checkout_cone; +extern int sparse_expect_files_outside_of_patterns;/* * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
config.c +14 -0
@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return platform_core_config (var, value, cb); } +static int git_default_sparse_config (const char *var,const char *value) +{ +if (!strcmp (var,"sparse.expectfilesoutsideofpatterns" )) { + sparse_expect_files_outside_of_patterns =git_config_bool (var, value); +return 0 ; + } + +/* Add other config variables here and to Documentation/config/sparse.txt. */ +return 0 ; +} +static int git_default_i18n_config (const char *var,const char *value) {if (!strcmp (var,"i18n.commitencoding" ))
@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)return 0 ; } +if (starts_with (var,"sparse." )) +return git_default_sparse_config (var, value); +/* Add other config variables here and to Documentation/config.txt. */ return 0 ; }
environment.c +1 -0
@@ -70,6 +70,7 @@ char *notes_ref_name;int grafts_replace_parents =1 ;int core_apply_sparse_checkout;int core_sparse_checkout_cone; +int sparse_expect_files_outside_of_patterns;int merge_log_config =-1 ;int precomposed_unicode =-1 ;/* see probe_utf8_pathname_composition() */ unsigned long pack_size_limit_cfg;
sparse-index.c +2 -1
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)int i; -if (!core_apply_sparse_checkout) +if (!core_apply_sparse_checkout || + sparse_expect_files_outside_of_patterns)return ; restart:
t/t1090-sparse-checkout-scope.sh +19 -0
@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' ' test "$(cat b)" = "modified" ' +test_expect_success 'skip-worktree on files outside sparse patterns' ' + git sparse-checkout disable && + git sparse-checkout set --no-cone "a*" && + git checkout-index --all --ignore-skip-worktree-bits && + + git ls-files -t >output && + ! grep ^S output >actual && + test_must_be_empty actual && + + test_config sparse.expectFilesOutsideOfPatterns true && + cat <<-\EOF >expect && + S b + S c + EOF + git ls-files -t >output && + grep ^S output >actual && + test_cmp expect actual +' + test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' ' test_create_repo server && git clone "file://$(pwd)/server" client &&
Signature · 1 line
2.35.1.278.g4f05fa7020.dirty
PATCH v5 1/1 repo_read_index: add config to expect files outside sparse patterns
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as SKIP_WORKTREE and be missing from the working tree. Sometimes this expectation would be violated however; including in cases such as:
* users grabbing files from elsewhere and writing them to the worktree (perhaps by editing a cached copy in an editor, copying/renaming, or even untarring)
* various git commands having incomplete or no support for the
SKIP_WORKTREE bit[1,2]
* users attempting to "abort" a sparse-checkout operation with a not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the working tree is not atomic)[3].
When the SKIP_WORKTREE bit in the index did not reflect the presence of the file in the working tree, it traditionally caused confusion and was difficult to detect and recover from. So, in a sparse checkout, since af6a51875a ("repo_read_index: clear SKIP_WORKTREE bit from files present in worktree", 2022-01-06), Git automatically clears the SKIP_WORKTREE bit at index read time for entries corresponding to files that are present in the working tree.
There is another workflow, however, where it is expected that paths outside the sparsity patterns appear to exist in the working tree and that they do not lose the SKIP_WORKTREE bit, at least until they get modified. A Git-aware virtual file system[4] takes advantage of its position as a file system driver to expose all files in the working tree, fetch them on demand using partial clone on access, and tell Git to pay attention to them on demand by updating the sparse checkout pattern on writes. This means that commands like "git status" only have to examine files that have potentially been modified, whereas commands like "ls" are able to show the entire codebase without requiring manual updates to the sparse checkout pattern.
Thus since af6a51875a, Git with such Git-aware virtual file systems unsets the SKIP_WORKTREE bit for all files and commands like "git status" have to fetch and examine them all.
Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to allow limiting the tracked set of files to a small set once again. A Git-aware virtual file system or other application that wants to maintain files outside of the sparse checkout can set this in a repository to instruct Git not to check for the presence of SKIP_WORKTREE files. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from the variety of difficult issues detailed in af6a51875a for paths with SKIP_WORKTREE set despite the path being present.
[1] https://lore.kernel.org/git/[email protected]/
[2] The three long paragraphs in the middle of
https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
[4] such as the vfsd described in
https://lore.kernel.org/git/[email protected]/
---
Changes since v4:
- Corrected an incorrect footnote
Range-diff:
1: 5ded1532cd ! 1: d348cc766f repo_read_index: add config to expect files outside sparse patterns
@@ Commit message
[2] The three long paragraphs in the middle of
https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
- [1] such as the vfsd described in
+ [4] such as the vfsd described in
https://lore.kernel.org/git/[email protected]/
Documentation/config.txt | 2 ++ Documentation/config/sparse.txt | 27 +++++++++++++++++++++++++++ cache.h | 1 + config.c | 14 ++++++++++++++ environment.c | 1 + sparse-index.c | 3 ++- t/t1090-sparse-checkout-scope.sh | 19 +++++++++++++++++++ 7 files changed, 66 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/sparse.txt
Documentation/config.txt +2 -0
@@ -468,6 +468,8 @@ include::config/sequencer.txt[] include::config/showbranch.txt[] +include::config/sparse.txt[] + include::config/splitindex.txt[] include::config/ssh.txt[]
Documentation/config/sparse.txt new file +27 -0
@@ -0,0 +1,27 @@ +sparse.expectFilesOutsideOfPatterns:: + Typically with sparse checkouts, files not matching any + sparsity patterns are marked with a SKIP_WORKTREE bit in the + index and are missing from the working tree. Accordingly, Git + will ordinarily check whether files with the SKIP_WORKTREE bit + are in fact present in the working tree contrary to + expectations. If Git finds any, it marks those paths as + present by clearing the relevant SKIP_WORKTREE bits. This + option can be used to tell Git that such + present-despite-skipped files are expected and to stop + checking for them. ++ +The default is `false`, which allows Git to automatically recover +from the list of files in the index and working tree falling out of +sync. ++ +Set this to `true` if you are in a setup where some external factor +relieves Git of the responsibility for maintaining the consistency +between the presence of working tree files and sparsity patterns. For +example, if you have a Git-aware virtual file system that has a robust +mechanism for keeping the working tree and the sparsity patterns up to +date based on access patterns. ++ +Regardless of this setting, Git does not check for +present-despite-skipped files unless sparse checkout is enabled, so +this config option has no effect unless `core.sparseCheckout` is +`true`.
cache.h +1 -0
@@ -1003,6 +1003,7 @@ extern const char *core_fsmonitor;extern int core_apply_sparse_checkout;extern int core_sparse_checkout_cone; +extern int sparse_expect_files_outside_of_patterns;/* * Returns the boolean value of $GIT_OPTIONAL_LOCKS (or the default value).
config.c +14 -0
@@ -1544,6 +1544,17 @@ static int git_default_core_config(const char *var, const char *value, void *cb)return platform_core_config (var, value, cb); } +static int git_default_sparse_config (const char *var,const char *value) +{ +if (!strcmp (var,"sparse.expectfilesoutsideofpatterns" )) { + sparse_expect_files_outside_of_patterns =git_config_bool (var, value); +return 0 ; + } + +/* Add other config variables here and to Documentation/config/sparse.txt. */ +return 0 ; +} +static int git_default_i18n_config (const char *var,const char *value) {if (!strcmp (var,"i18n.commitencoding" ))
@@ -1675,6 +1686,9 @@ int git_default_config(const char *var, const char *value, void *cb)return 0 ; } +if (starts_with (var,"sparse." )) +return git_default_sparse_config (var, value); +/* Add other config variables here and to Documentation/config.txt. */ return 0 ; }
environment.c +1 -0
@@ -70,6 +70,7 @@ char *notes_ref_name;int grafts_replace_parents =1 ;int core_apply_sparse_checkout;int core_sparse_checkout_cone; +int sparse_expect_files_outside_of_patterns;int merge_log_config =-1 ;int precomposed_unicode =-1 ;/* see probe_utf8_pathname_composition() */ unsigned long pack_size_limit_cfg;
sparse-index.c +2 -1
@@ -396,7 +396,8 @@ void clear_skip_worktree_from_present_files(struct index_state *istate)int i; -if (!core_apply_sparse_checkout) +if (!core_apply_sparse_checkout || + sparse_expect_files_outside_of_patterns)return ; restart:
t/t1090-sparse-checkout-scope.sh +19 -0
@@ -52,6 +52,25 @@ test_expect_success 'return to full checkout of main' ' test "$(cat b)" = "modified" ' +test_expect_success 'skip-worktree on files outside sparse patterns' ' + git sparse-checkout disable && + git sparse-checkout set --no-cone "a*" && + git checkout-index --all --ignore-skip-worktree-bits && + + git ls-files -t >output && + ! grep ^S output >actual && + test_must_be_empty actual && + + test_config sparse.expectFilesOutsideOfPatterns true && + cat <<-\EOF >expect && + S b + S c + EOF + git ls-files -t >output && + grep ^S output >actual && + test_cmp expect actual +' + test_expect_success 'in partial clone, sparse checkout only fetches needed blobs' ' test_create_repo server && git clone "file://$(pwd)/server" client &&
Signature · 1 line
2.35.1.355.g49d31b1b5a.dirty
Re: [PATCH v5] repo_read_index: add config to expect files outside sparse patterns
Quoted 9 lines · Elijah Newren
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as
SKIP_WORKTREEand be missing from the working tree. Sometimes this expectation would be violated however; including in cases such as:
* users grabbing files from elsewhere and writing them to the worktree (perhaps by editing a cached copy in an editor, copying/renaming, or even untarring)
* various git commands having incomplete or no support for theSKIP_WORKTREE bit[1,2]
Quoted 8 lines
* users attempting to "abort" a sparse-checkout operation with a not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the working tree is not atomic)[3].
When theSKIP_WORKTREEbit in the index did not reflect the presence of the file in the working tree, it traditionally caused confusion and was difficult to detect and recover from. So, in a sparse checkout, sinceaf6a51875a("repo_read_index: clearSKIP_WORKTREEbit from files present in worktree", 2022-01-06), Git automatically clears theSKIP_WORKTREE
The reference is a bit off here. Here is what I get locally:
af6a51875a (repo_read_index: clear SKIP_WORKTREE bit from files present in worktree, 2022-01-14)
and that is in the version I have locally in 'next'.
Quoted 5 lines
[1] https://lore.kernel.org/git/[email protected]/
[2] The three long paragraphs in the middle of
https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
[4] such as the vfsd described in
Here is another difference from the version I have locally in 'next', which I didn't notice that this [4] was misspelt as [1] before applying.
Everything else seems the same, so let's not bother reverting the old one out of 'next' and merging this version after fixing this version up. What we have is good enough modulo [4] vs [1].
Thanks.
Re: [PATCH v5] repo_read_index: add config to expect files outside sparse patterns
Quoted 21 lines · Junio C Hamano
Quoted 9 lines · Elijah Newren
Typically with sparse checkouts, we expect files outside the sparsity patterns to be marked as
SKIP_WORKTREEand be missing from the working tree. Sometimes this expectation would be violated however; including in cases such as:
* users grabbing files from elsewhere and writing them to the worktree (perhaps by editing a cached copy in an editor, copying/renaming, or even untarring)
* various git commands having incomplete or no support for theSKIP_WORKTREE bit[1,2]Quoted 8 lines
* users attempting to "abort" a sparse-checkout operation with a not-so-early Ctrl+C (updating $GIT_DIR/info/sparse-checkout and the working tree is not atomic)[3].
When theSKIP_WORKTREEbit in the index did not reflect the presence of the file in the working tree, it traditionally caused confusion and was difficult to detect and recover from. So, in a sparse checkout, sinceaf6a51875a("repo_read_index: clearSKIP_WORKTREEbit from files present in worktree", 2022-01-06), Git automatically clears theSKIP_WORKTREE
Quoted 6 lines
The reference is a bit off here. Here is what I get locally:
af6a51875a(repo_read_index: clearSKIP_WORKTREEbit from files present in worktree, 2022-01-14)and that is in the version I have locally in 'next'.
Ugh, forgot to update the date when I updated the reference when you pointed that out.
Quoted 9 lines
Quoted 5 lines
[1] https://lore.kernel.org/git/[email protected]/
[2] The three long paragraphs in the middle of
https://lore.kernel.org/git/CABPp-BH9tju7WVm=QZDOvaMDdZbpNXrVWQdN-jmfN8wC6YVhmw@mail.gmail.com/
[3] https://lore.kernel.org/git/CABPp-BFnFpzwGC11TLoLs8YK5yiisA5D5-fFjXnJsbESVDwZsA@mail.gmail.com/
[4] such as the vfsd described inHere is another difference from the version I have locally in 'next', which I didn't notice that this [4] was misspelt as [1] before applying.
Sorry, I hadn't noticed you merging to next, and I saw in the irc logs the discussion about this 1 vs. 4 between you and jrnieder so I thought I'd fix it.
Quoted 3 lines
Everything else seems the same, so let's not bother reverting the old one out of 'next' and merging this version after fixing this version up. What we have is good enough modulo [4] vs [1].
Sounds good.
Re: [PATCH v4] repo_read_index: add config to expect files outside sparse patterns
In reply to Elijah NewrenQuoted 10 lines · Elijah Newren
Introduce a configuration setting sparse.expectFilesOutsideOfPatterns to allow limiting the tracked set of files to a small set once again. A Git-aware virtual file system or other application that wants to maintain files outside of the sparse checkout can set this in a repository to instruct Git not to check for the presence of
SKIP_WORKTREEfiles. The setting defaults to false, so most users of sparse checkout will still get the benefit of an automatically updating index to recover from the variety of difficult issues detailed inaf6a51875afor paths withSKIP_WORKTREEset despite the path being present.
Sorry that I lost track of this thread recently. I re-read this version that made it to 'next' already and have no issues.
This approach will work well for our needs in the microsoft/git fork. Likely, we'll enable the sparse_expect_files_outside_of_patterns global when we initialize our core_virtualfilesystem global. Then, we'll probably replace some of the uses of core_virtualfilesytem with this new global in some of the conditions that we inserted to work with previous restrictions in this area. Then, those changes will be fit for submitting here. If anyone needs those changes sooner, then do not hesitate to make similar changes. We will adapt in our fork to whatever the Git community thinks is best.
Thanks,
-Stolee