Bug report - local (and git ignored) file silently removed after checkout

Context:
- in previous version of my project, the TODO file was versioned
- in current development version, this file is NO more versioned AND
it is ignored by a gitignore rule,...

We do not have "not tracked, but precious (e.g. your TODO)" category; from time to time we have discussed possibilities of adding such to enrich the categorization, but nobody stepped up to actually do the work of designing let alone coding such a beast.

Junio C Hamano, Sun 2011-11-20 @ 13:16:27-0800:

Quoted 4 lines

We do not have "not tracked, but precious (e.g. your TODO)" category; from time to time we have discussed possibilities of adding such to enrich the categorization, but nobody stepped up to actually do the work of designing let alone coding such a beast.

If I'm not mistaken, files ignored by .git/info/exclude rather than .gitignore do exhibit this behavior.

As far as I am aware, info/exclude should work exactly the same as having a .gitignore file at the root level of the working tree. Can you show a minimum reproduction recipe in a form of a patch to our test scripts in t/ hierarchy?

From: Johannes Sixt <[email protected]>

It is an unintended accident that entries matched by .git/info/exclude are considered precious, but entries matched by .gitignore are not. That is, 'git checkout' will overwrite untracked files matched by .gitignore, but refuses to overwrite files matched by .git/info/exclude.

It is a lucky accident: it allows the distinction between "untracked but precious" and "untracked and garbage". And it is a doubly lucky accident:
.gitignore entries are meant for files like build products, which usually affect all consumers of a repository, whereas .git/info/exclude is intended for personal files, which frequently are precious (think of a TODO file).

Add a test that codifies the accident as wanted behavior.

Signed-off-by Johannes Sixt <[email protected]>

---
Am 11/21/2011 4:36, schrieb Junio C Hamano:

Quoted 4 lines

As far as I am aware, info/exclude should work exactly the same as having a .gitignore file at the root level of the working tree. Can you show a minimum reproduction recipe in a form of a patch to our test scripts in t/ hierarchy?

Here you are. As you can see from my commit message, IMO, this is a very useful accident. Therefore, there is no 'test_expect_failure' in the test script :-)

-- Hannes

t/t2023-checkout-ignored.sh | 51
+++++++++++++++++++++++++++++++++++++++++++

1 files changed, 51 insertions(+), 0 deletions(-)

create mode 100755 t/t2023-checkout-ignored.sh

t/t2023-checkout-ignored.sh new file +51 -0

new file mode 100755
index 0000000..03a5a56
@@ -0,0 +1,51 @@
+#!/bin/sh
+
+test_description='checkout overwrites or preserves ignored files
+
+`git checkout` makes a distinction between files mentioned in
+.gitignore and .git/info/exclude in that untracked files matched
+by the latter are considered precious and are not overwritten.
+'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+
+ echo excluded > excluded &&
+ echo ignored > ignored &&
+ git add . &&
+ test_commit initial &&
+ git checkout -b side &&
+ git rm excluded &&
+ git mv ignored .gitignore &&
+ test_commit side &&
+ echo excluded >> .git/info/exclude
+'
+
+test_expect_success 'files are ignored' '
+
+ echo keep > excluded &&
+ echo overwrite > ignored &&
+ list=$(git ls-files --others --exclude-standard) &&
+ test -z "$list"
+'
+
+test_expect_success 'entries in .git/info/exclude are precious' '
+
+ test_must_fail git checkout master 2>errors &&
+ test_i18ngrep "would be overwritten" errors &&
+ grep " excluded" errors &&
+ ! grep " ignored" errors &&
+ grep keep excluded &&
+ grep overwrite ignored
+'
+
+test_expect_success 'entries in .gitignore are not precious' '
+
+ rm -f excluded &&
+ git checkout master &&
+ grep excluded excluded &&
+ grep ignored ignored
+'
+
+test_done

Signature · 11 lines

1.7.8.rc0.126.gd15506
--
To unsubscribe from this list: send the line "unsubscribe git" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html


-----
No virus found in this message.
Checked by AVG - www.avg.com
Version: 2012.0.1872 / Virus Database: 2092/4628 - Release Date: 11/20/11

Shouldn't there be some documentation changes to support this very useful feature.

From: Johannes Sixt <[email protected]>

It is an unintended accident that entries matched by .git/info/exclude are considered precious, but entries matched by .gitignore are not. That is, 'git checkout' will overwrite untracked files matched by .gitignore, but refuses to overwrite files matched by .git/info/exclude.

It is a lucky accident: it allows the distinction between "untracked but precious" and "untracked and garbage". And it is a doubly lucky accident:
.gitignore entries are meant for files like build products, which usually affect all consumers of a repository, whereas .git/info/exclude is intended for personal files, which frequently are precious (think of a TODO file).

Add a test that codifies the accident as wanted behavior.

If you want to keep this accident (which is a bug to me), you may want to add the reason: callers to unpack_trees() are supposed to also setup exclude rules in .git/info/exclude before calling unpack_trees(), which they don't.

So .git/info/exclude is entirely dismissed.

Quoted 6 lines · Nguyen Thai Ngoc Duy

If you want to keep this accident (which is a bug to me), you may want to add the reason: callers to unpack_trees() are supposed to also setup exclude rules in .git/info/exclude before calling unpack_trees(), which they don't.

So .git/info/exclude is entirely dismissed.

Ohh, thanks for this and also for digging this through in your other message.

I think it is the right thing to do to make sure .gitignore and info/exclude behave the same way regardless of the original issue in this topic.

In the medium term, I think one reasonable way forward solving the "TODO that used to be tracked but now untracked and ignored" issue is to introduce "info/exclude-override" that comes between command line and in-tree patterns. The info/exclude file is designed as the fallback definition to be used when all other sources are too lax, and comes near the precedence stack; the "TODO" situation however calls for an override that is stronger than the in-tree patterns.

In the longer term, we should carefully determine if we need "precious" in the first place. The last time this was brought up there were people who argued they are OK with having to remove the ignored file by hand when checking out another branch (i.e. we switch the semantics of "ignored" so that they are "not tracked but all precious").

I think it matters in two cases.

 (1) If you change an untracked "cruft" file on branch A into a directory with tracked files in it on another branch B. If you are on branch A, have that "cruft" file (perhaps it is a build product after running "make"), and try to checkout branch B, such an updated "git checkout" will start erroring out telling you that "cruft" will be lost.

 (2) If you have a directory on branch A, underneath of which there are untracked "cruft" files (e.g. think "build/" directory that is full of "*.o" files and ".gitignore" to mark object files as ignored but is otherwise empty), and another branch B that has the same path as a file. If you are on branch A, have "cruft" files in that directory, and try to checkout branch B, such an updated "git checkout" will start erroring out telling you that "cruft" will be lost.

If people are OK with such a behaviour, we can do without "precious".

Otherwise we would need to update excluded() in dir.c to return tristate (ignored, precious or unspecified) instead of the current boolean (ignored or unspecified), examine and decide for each caller what they want to do to "precious" files.

In the medium term, I think one reasonable way forward solving the "TODO that used to be tracked but now untracked and ignored" issue is to introduce "info/exclude-override" that comes between command line and in-tree patterns. The info/exclude file is designed as the fallback definition to be used when all other sources are too lax, and comes near the precedence stack; the "TODO" situation however calls for an override that is stronger than the in-tree patterns.

In the longer term, we should carefully determine if we need "precious" in the first place. The last time this was brought up there were people who argued they are OK with having to remove the ignored file by hand when checking out another branch (i.e. we switch the semantics of "ignored" so that they are "not tracked but all precious").

I think it matters in two cases.

 (1) If you change an untracked "cruft" file on branch A into a directory with tracked files in it on another branch B. If you are on branch A, have that "cruft" file (perhaps it is a build product after running "make"), and try to checkout branch B, such an updated "git checkout" will start erroring out telling you that "cruft" will be lost.

 (2) If you have a directory on branch A, underneath of which there are untracked "cruft" files (e.g. think "build/" directory that is full of "*.o" files and ".gitignore" to mark object files as ignored but is otherwise empty), and another branch B that has the same path as a file. If you are on branch A, have "cruft" files in that directory, and try to checkout branch B, such an updated "git checkout" will start erroring out telling you that "cruft" will be lost.

If people are OK with such a behaviour, we can do without "precious".

Otherwise we would need to update excluded() in dir.c to return tristate (ignored, precious or unspecified) instead of the current boolean (ignored or unspecified), examine and decide for each caller what they want to do to "precious" files.

Quoted 7 lines · Junio C Hamano

In the medium term, I think one reasonable way forward solving the "TODO that used to be tracked but now untracked and ignored" issue is to introduce "info/exclude-override" that comes between command line and in-tree patterns. The info/exclude file is designed as the fallback definition to be used when all other sources are too lax, and comes near the precedence stack; the "TODO" situation however calls for an override that is stronger than the in-tree patterns.

"info/precious" might be a better name

Quoted 14 lines

In the longer term, we should carefully determine if we need "precious" in the first place. The last time this was brought up there were people who argued they are OK with having to remove the ignored file by hand when checking out another branch (i.e. we switch the semantics of "ignored" so that they are "not tracked but all precious").

I think it matters in two cases.

 (1) If you change an untracked "cruft" file on branch A into a directory with tracked files in it on another branch B. If you are on branch A, have that "cruft" file (perhaps it is a build product after running "make"), and try to checkout branch B, such an updated "git checkout" will start erroring out telling you that "cruft" will be lost.

Quoted 7 lines

 (2) If you have a directory on branch A, underneath of which there are untracked "cruft" files (e.g. think "build/" directory that is full of "*.o" files and ".gitignore" to mark object files as ignored but is otherwise empty), and another branch B that has the same path as a file. If you are on branch A, have "cruft" files in that directory, and try to checkout branch B, such an updated "git checkout" will start erroring out telling you that "cruft" will be lost.

I think we should do this regardless precious being added or not.

Quoted 1 line

If people are OK with such a behaviour, we can do without "precious".

What about git-clean to remove ignored but not precious files?

Or do excluded() twice, the first time to check for precious files, the second for all ignored rules. Callers are changed anyway, but this way git-add for example will be untouched because it does not care about precious stuff. Only unpack-trees and maybe git-clean are changed.

Quoted 9 lines · Nguyen Thai Ngoc Duy

Quoted 7 lines · Junio C Hamano

In the medium term, I think one reasonable way forward solving the "TODO that used to be tracked but now untracked and ignored" issue is to introduce "info/exclude-override" that comes between command line and in-tree patterns. The info/exclude file is designed as the fallback definition to be used when all other sources are too lax, and comes near the precedence stack; the "TODO" situation however calls for an override that is stronger than the in-tree patterns.

Quoted 1 line

"info/precious" might be a better name

The above is only about the precedence order and is not about introducing the new "precious" class at all.

Quoted 22 lines

Quoted 21 lines

In the longer term, we should carefully determine if we need "precious" in the first place. The last time this was brought up there were people who argued they are OK with having to remove the ignored file by hand when checking out another branch (i.e. we switch the semantics of "ignored" so that they are "not tracked but all precious").

I think it matters in two cases.

 (1) If you change an untracked "cruft" file on branch A into a directory with tracked files in it on another branch B. If you are on branch A, have that "cruft" file (perhaps it is a build product after running "make"), and try to checkout branch B, such an updated "git checkout" will start erroring out telling you that "cruft" will be lost.

 (2) If you have a directory on branch A, underneath of which there are untracked "cruft" files (e.g. think "build/" directory that is full of "*.o" files and ".gitignore" to mark object files as ignored but is otherwise empty), and another branch B that has the same path as a file. If you are on branch A, have "cruft" files in that directory, and try to checkout branch B, such an updated "git checkout" will start erroring out telling you that "cruft" will be lost.

Quoted 1 line

I think we should do this regardless precious being added or not.

Because (see below)?

Quoted 3 lines

Quoted 1 line

If people are OK with such a behaviour, we can do without "precious".

What about git-clean to remove ignored but not precious files?

"clean -x" is more subtle. It has been a way to say "Remove cruft the usual way, and in addition, remove the expendable build products, just like 'make clean' _should_ do, but I do not trust my Makefile". If we introduced "precious", it would be very clear what it should do---even with "-x" precious files should be kept. But if we don't and just try to get away by changing the semantics of "ignored", they will still need to be removed, so we won't really get the "precious".

The conclusion from this is that it is a mistake to change the semantics of "ignored" from the current "untracked and expendable if needed" if the purpose of that change is to avoid introducing the new "precious" class.

I don't care too much about it, as I do not use "git clean -x" myself ;-) but that wouldn't stop others from think about the issue and try to come up with a good solid design.

...

Quoted 3 lines

Quoted 1 line

I think we should do this regardless precious being added or not.

Because (see below)?

Because it may potentially lose user's changes. Assume file "A" is tracked and also ignored. Users may make some changes in A, then move HEAD away without touching worktree, now HEAD does not see "A" as tracked any more.

Quoted 9 lines · Nguyen Thai Ngoc Duy

...

Quoted 3 lines

Quoted 1 line

I think we should do this regardless precious being added or not.

Because (see below)?

Because it may potentially lose user's changes. Assume file "A" is tracked and also ignored. Users may make some changes in A, then move HEAD away without touching worktree, now HEAD does not see "A" as tracked any more.

Huh? "A" is initially tracked and the user modifies it. Moving HEAD away how?  "git checkout" would refuse to check out a branch that has different contents at "A", or is missing "A". So how can "now HEAD does not see 'A' as tracked any more" happen?

In the medium term, I think one reasonable way forward solving the "TODO that used to be tracked but now untracked and ignored" issue is to introduce "info/exclude-override" that comes between command line and in-tree patterns. The info/exclude file is designed as the fallback definition to be used when all other sources are too lax, and comes near the precedence stack; the "TODO" situation however calls for an override that is stronger than the in-tree patterns.

Back in 1127148 (Loosen "working file will be lost" check in Porcelain-ish - 2006-12-04), git-checkout.sh learned to quietly overwrite ignored files. Howver the code only took .gitignore files into account.

It is a lucky accident: it allows the distinction between "untracked but precious" and "untracked and garbage". And it is a doubly lucky accident:
.gitignore entries are meant for files like build products, which usually affect all consumers of a repository, whereas .git/info/exclude is intended for personal files, which frequently are precious (think of a TODO file).
...
Here you are. As you can see from my commit message, IMO, this is a very useful accident. Therefore, there is no 'test_expect_failure' in the test script :-)

If this is a feature, we would like a patch to Documentation/gitignore.txt as well, I think.  Also I have a suspicion that this was an unintended "regression"; do you have a bisection?

It dated back to f8a9d42 (read-tree: further loosen "working file will be lost" check. - 2006-12-04) when you introduced --exclude-per-directory to read-tree, but not --exclude-from to explicitly add .git/info/exclude.

I guess it's time to add "read-tree --exclude-from".

We may add an option to checkout to skip
either .git/info/exclude or .gitignore, or both.

Quoted 3 lines · Nguyen Thai Ngoc Duy

It dated back to f8a9d42 (read-tree: further loosen "working file will be lost" check. - 2006-12-04) when you introduced --exclude-per-directory to read-tree, but not --exclude-from to explicitly add .git/info/exclude.

That one together with 1127148 (Loosen "working file will be lost" check in Porcelain-ish, 2006-12-04) complets the picture. Thanks for digging this out.

Quoted 1 line

I guess it's time to add "read-tree --exclude-from".

Perhaps, also --exclude-standard to match?