"add worktree" fails with "fatal: Invalid path" error
Hi,
I'm running git v.2.20.1 on Ubuntu from a program which follows the pattern:
============
1. create a temporary directory /tmp/tmpabc
2. in a loop:
2a. create a second level of temporary directory /tmp/tmpabc/tmpworktree123 2b. use "git worktree add" on the second level directory
2c. do something
3. cleanup
3b. "git branch -D" on each basename(second level directory) 3a. "git worktree prune"
============
The loop size is of the order of 8-20. In step 2b, I often get errors like this (from a Bash reproducer):
============
$ git worktree add /tmp/tmpgtxug4y9/git_worktree.gBGqnfnU Preparing worktree (new branch 'git_worktree.gBGqnfnU') fatal: Invalid path '/tmp/tmp1q9ysvyl': No such file or directory ============
I can see that the problematic path exists in the "gitdir" file of what must be an earlier worktree from an older run (the branch is gone, but the tree is still there). The path appear to relate to the older run's first level directory:
============
$ grep -r /tmp/tmp1q9ysvyl ../.git/worktrees/ ../.git/worktrees/git_worktree.frcwtjt_/gitdir:/tmp/tmp1q9ysvyl/git_worktree.frcwtjt_/.git $ git worktree list ... /tmp/tmp1q9ysvyl/git_worktree.frcwtjt_ edde3f25 (detached HEAD) ... $ git branch | grep frcwtjt_ <no matches> ============
NOTE: I've not yet had to try deleting the worktree, since "add worktree" does appear to work some of the time, so I am able to limp along.
I have these questions:
1. There is no branch or first level directory, but "git prune" has not deleted the worktree, is this expected?
2. Is there something wrong with the sequence of steps I am following?
Thanks, Shaheed
P.S. I have an strace of a failing worktree add if needed.
Quoted 7 lines · Shaheed Haque
Hi,
I'm running git v.2.20.1 on Ubuntu from a program which follows the pattern:
============
1. create a temporary directory/tmp/tmpabc
When is this directory deleted? After step 3a?
Quoted 24 lines
2. in a loop:
2a. create a second level of temporary directory/tmp/tmpabc/tmpworktree1232b. use "git worktree add" on the second level directory
2c. do something
3. cleanup
3b. "git branch -D" on each basename(second level directory) 3a. "git worktree prune"
============The loop size is of the order of 8-20. In step 2b, I often get errors like this (from a Bash reproducer):
============
$ git worktree add /tmp/tmpgtxug4y9/git_worktree.gBGqnfnU Preparing worktree (new branch 'git_worktree.gBGqnfnU') fatal: Invalid path '/tmp/tmp1q9ysvyl': No such file or directory ============I can see that the problematic path exists in the "gitdir" file of what must be an earlier worktree from an older run (the branch is gone, but the tree is still there). The path appear to relate to the older run's first level directory:
============
Quoted 9 lines
$ grep -r /tmp/tmp1q9ysvyl ../.git/worktrees/ ../.git/worktrees/git_worktree.frcwtjt_/gitdir:/tmp/tmp1q9ysvyl/git_worktree.frcwtjt_/.git $ git worktree list ... /tmp/tmp1q9ysvyl/git_worktree.frcwtjt_ edde3f25 (detached HEAD) ... $ git branch | grep frcwtjt_ <no matches> ============
Yeah I think I know where that "Invalid path" comes from and it should not be there (at least it should not be a fatal error). I'll need to reproduce this first. But I'm certain you've given me enough information to do so.
Quoted 3 lines
NOTE: I've not yet had to try deleting the worktree, since "add worktree" does appear to work some of the time, so I am able to limp along.
It's probably best to stay clean and delete things after you're done. At least you should be able to avoid this problem this way until it's fixed.
Quoted 5 lines
I have these questions:
1. There is no branch or first level directory, but "git prune" has not deleted the worktree, is this expected?
I assume you meant "git worktree prune", not "git prune". See gc.worktreePruneExpire. Dead worktree info stays for a while until it's deleted, so that you can recover stuff if you need to.
Quoted 1 line
2. Is there something wrong with the sequence of steps I am following?
Nope. I mean, you could try "git worktree remove" to be on the safe side. But it should work even without that. To me this looks very much like a bug.
Quoted 3 lines
Thanks, Shaheed
P.S. I have an strace of a failing worktree add if needed.
Signature · 1 line
Duy
Hi Duy,
Quoted 11 lines · Duy Nguyen
Quoted 7 lines · Shaheed Haque
Hi,
I'm running git v.2.20.1 on Ubuntu from a program which follows the pattern:
============
1. create a temporary directory/tmp/tmpabcWhen is this directory deleted? After step 3a?
Yes. I note that I screwed up the numbering in my note - the order is ["git branch -D", "git worktree prune", delete directory].
Quoted 39 lines
Quoted 24 lines
2. in a loop:
2a. create a second level of temporary directory/tmp/tmpabc/tmpworktree1232b. use "git worktree add" on the second level directory
2c. do something
3. cleanup
3b. "git branch -D" on each basename(second level directory) 3a. "git worktree prune"
============The loop size is of the order of 8-20. In step 2b, I often get errors like this (from a Bash reproducer):
============
$ git worktree add /tmp/tmpgtxug4y9/git_worktree.gBGqnfnU Preparing worktree (new branch 'git_worktree.gBGqnfnU') fatal: Invalid path '/tmp/tmp1q9ysvyl': No such file or directory ============I can see that the problematic path exists in the "gitdir" file of what must be an earlier worktree from an older run (the branch is gone, but the tree is still there). The path appear to relate to the older run's first level directory:
============
Quoted 9 lines
$ grep -r /tmp/tmp1q9ysvyl ../.git/worktrees/ ../.git/worktrees/git_worktree.frcwtjt_/gitdir:/tmp/tmp1q9ysvyl/git_worktree.frcwtjt_/.git $ git worktree list ... /tmp/tmp1q9ysvyl/git_worktree.frcwtjt_ edde3f25 (detached HEAD) ... $ git branch | grep frcwtjt_ <no matches> ============Yeah I think I know where that "Invalid path" comes from and it should not be there (at least it should not be a fatal error). I'll need to reproduce this first. But I'm certain you've given me enough information to do so.
Quoted 7 lines
Quoted 3 lines
NOTE: I've not yet had to try deleting the worktree, since "add worktree" does appear to work some of the time, so I am able to limp along.
It's probably best to stay clean and delete things after you're done. At least you should be able to avoid this problem this way until it's fixed.
Ack.
Quoted 8 lines
Quoted 4 lines
I have these questions:
1. There is no branch or first level directory, but "git prune" has not deleted the worktree, is this expected?
I assume you meant "git worktree prune", not "git prune". See gc.worktreePruneExpire. Dead worktree info stays for a while until it's deleted, so that you can recover stuff if you need to.
Yes, sorry for typo.
Quoted 5 lines
Quoted 1 line
2. Is there something wrong with the sequence of steps I am following?
Nope. I mean, you could try "git worktree remove" to be on the safe side. But it should work even without that. To me this looks very much like a bug.
The original code used the more obvious "git worktree remove" rather than "git worktree prune" but I switched partly because remove seemed slow (I cannot now quantify what caused me to think that), and partly because I was having other issues which, I now realise, you probably addressed in your recent "stat versus mkdir race" change.
BTW: I *love* worktrees. I have used/wrapped/developed multiple source control systems over the years, and IMHO, this is one of the killer features of git.
Thanks, Shaheed
Quoted 5 lines
Quoted 3 lines
Thanks, Shaheed
P.S. I have an strace of a failing worktree add if needed.--
Duy
Quoted 5 lines · Shaheed Haque
The original code used the more obvious "git worktree remove" rather than "git worktree prune" but I switched partly because remove seemed slow (I cannot now quantify what caused me to think that), and partly because I was having other issues which, I now realise, you probably addressed in your recent "stat versus mkdir race" change.
It should be as slow as "git status; rm -r". The first command _could_ be slow. But if you find it significantly slower than that, I will be glad to receive another bug report.
Signature · 1 line
Duy
Quoted 11 lines · Duy Nguyen
Quoted 5 lines · Shaheed Haque
The original code used the more obvious "git worktree remove" rather than "git worktree prune" but I switched partly because remove seemed slow (I cannot now quantify what caused me to think that), and partly because I was having other issues which, I now realise, you probably addressed in your recent "stat versus mkdir race" change.
It should be as slow as "git status; rm -r". The first command _could_ be slow. But if you find it significantly slower than that, I will be glad to receive another bug report.
After I wrote, I went back and checked, and I have no idea why I thought it slow. It seems just fine (by the time I have spawned out of Python and all), and so I have switched the code back to (synchronous/predictable) "git worktree remove".
Thanks, Shaheed
Quoted 2 lines
--
Duy
PATCH 1/1 worktree add: be tolerant of corrupt worktrees
In reply to Shaheed Haquefind_worktree() can die() unexpectedly because it uses real_path() instead of the gentler version. When it's used in 'git worktree add' [1] and there's a bad worktree, this die() could prevent people from adding new worktrees.
The "bad" condition to trigger this is when a parent of the worktree's location is deleted. Then real_path() will complain.
Use the other version so that bad worktrees won't affect 'worktree add'. The bad ones will eventually be pruned, we just have to tolerate them for a bit.
[1] added in cb56f55c16 (worktree: disallow adding same path multiple times, 2018-08-28), or since v2.20.0. Though the real bug in find_worktree() is much older.
--- t/t2025-worktree-add.sh | 12 ++++++++++++ worktree.c | 7 +++++-- 2 files changed, 17 insertions(+), 2 deletions(-)
t/t2025-worktree-add.sh +12 -0
@@ -570,4 +570,16 @@ test_expect_success '"add" an existing locked but missing worktree' ' git worktree add --force --force --detach gnoo ' +test_expect_success '"add" should not fail because of another bad worktree' ' + git init add-fail && + ( + cd add-fail && + test_commit first && + mkdir sub && + git worktree add sub/to-be-deleted && + rm -rf sub && + git worktree add second + ) +' + test_done
worktree.c +5 -2
@@ -222,9 +222,12 @@ struct worktree *find_worktree(struct worktree **list,free (to_free);return NULL ; } -for (; *list; list++) -if (!fspathcmp (path,real_path ((*list)->path))) +for (; *list; list++) { +const char *wt_path =real_path_if_valid ((*list)->path); + +if (wt_path && !fspathcmp (path, wt_path))break ; + }free (path);free (to_free);return *list;
Signature · 1 line
2.21.0.1141.gd54ac2cb17
Re: [PATCH] worktree add: be tolerant of corrupt worktrees
Hi Nguyễn,
Thanks for the quick response. While I leave the code to the experts, I can confirm that restoring the missing directory (but no content in it) does allow "worktree add" to function again.
One point may be worth clarifying...
Quoted 6 lines · Nguyễn Thái Ngọc Duy
find_worktree()candie()unexpectedly because it usesreal_path()instead of the gentler version. When it's used in 'git worktree add' [1] and there's a bad worktree, thisdie()could prevent people from adding new worktrees.
Quoted 6 lines
The "bad" condition to trigger this is when a parent of the worktree's location is deleted. Then
real_path()will complain.Use the other version so that bad worktrees won't affect 'worktree add'. The bad ones will eventually be pruned, we just have to tolerate them for a bit.
...as I mentioned, from my experiments, trying a "worktree prune" did NOT resolve the issue for me. But since I don't know the logic that prune uses, there may have been some other reason for this.
Thanks again, Shaheed
Quoted 54 lines
[1] added in
cb56f55c16(worktree: disallow adding same path multiple times, 2018-08-28), or since v2.20.0. Though the real bug infind_worktree()is much older.Reported-by Shaheed Haque <[email protected]>Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>--- t/t2025-worktree-add.sh | 12 ++++++++++++ worktree.c | 7 +++++-- 2 files changed, 17 insertions(+), 2 deletions(-)t/t2025-worktree-add.sh +12
-0@@ -570,4 +570,16 @@ test_expect_success '"add" an existing locked but missing worktree' ' git worktree add --force --force --detach gnoo ' +test_expect_success '"add" should not fail because of another bad worktree' ' + git init add-fail && + ( + cd add-fail && + test_commit first && + mkdir sub && + git worktree add sub/to-be-deleted && + rm -rf sub && + git worktree add second + ) +' + test_doneworktree.c +5
-3@@ -222,9 +222,12 @@ struct worktree *find_worktree(struct worktree **list,free (to_free);return NULL ; } -for (; *list; list++) -if (!fspathcmp (path,real_path ((*list)->path))) +for (; *list; list++) { +const char *wt_path =real_path_if_valid ((*list)->path); + +if (wt_path && !fspathcmp (path, wt_path))break ; + }free (path);free (to_free);return *list; --2.21.0.1141.gd54ac2cb17
Re: [PATCH] worktree add: be tolerant of corrupt worktrees
In reply to Nguyễn Thái Ngọc DuyQuoted 4 lines · Nguyễn Thái Ngọc Duy
find_worktree()candie()unexpectedly because it usesreal_path()instead of the gentler version. When it's used in 'git worktree add' [1] and there's a bad worktree, thisdie()could prevent people from adding new worktrees.
This is good to know because, to fix [1], I think we'll want to add a new function[2] akin to find_worktree(), but without magic suffix matching (that is, just literal absolute path comparison).
[1]: https://public-inbox.org/git/[email protected]/
[2]: https://public-inbox.org/git/CAPig+cQh8hxeoVjLHDKhAcZVQPpPT5v0AUY8gsL9=qfJ7z-L2A@mail.gmail.com/
Quoted 6 lines
The "bad" condition to trigger this is when a parent of the worktree's location is deleted. Then
real_path()will complain.Use the other version so that bad worktrees won't affect 'worktree add'. The bad ones will eventually be pruned, we just have to tolerate them for a bit.
The patch itself makes sense, though, as Shaheed noted in his response, pruning seems to get short-circuited somehow under this situation; perhaps that needs its own fix, but certainly shouldn't hold up this fix.
Quoted 2 lines
Reported-by Shaheed Haque <[email protected]>Signed-off-by Nguyễn Thái Ngọc Duy <[email protected]>
Re: [PATCH] worktree add: be tolerant of corrupt worktrees
Quoted 10 lines · Eric Sunshine
Quoted 4 lines · Nguyễn Thái Ngọc Duy
find_worktree()candie()unexpectedly because it usesreal_path()instead of the gentler version. When it's used in 'git worktree add' [1] and there's a bad worktree, thisdie()could prevent people from adding new worktrees.This is good to know because, to fix [1], I think we'll want to add a new function[2] akin to
find_worktree(), but without magic suffix matching (that is, just literal absolute path comparison).
Yeah. find_worktree() was made to handle command line options from worktree's move/remove, it's probably a bit too magical for this case.
I still want to store relative path in "gitdir" files at some point, which would complicate the last "absolute path comparison" part a bit. But it should be manageable.
Quoted 6 lines
[1]: https://public-inbox.org/git/[email protected]/
[2]: https://public-inbox.org/git/CAPig+cQh8hxeoVjLHDKhAcZVQPpPT5v0AUY8gsL9=qfJ7z-L2A@mail.gmail.com/Quoted 3 lines
The "bad" condition to trigger this is when a parent of the worktree's location is deleted. Then
real_path()will complain.
Quoted 8 lines
Quoted 3 lines
Use the other version so that bad worktrees won't affect 'worktree add'. The bad ones will eventually be pruned, we just have to tolerate them for a bit.
The patch itself makes sense, though, as Shaheed noted in his response, pruning seems to get short-circuited somehow under this situation; perhaps that needs its own fix, but certainly shouldn't hold up this fix.
I might have missed that detail. Thanks for pointing out. Will get another look.
Signature · 1 line
Duy
Re: [PATCH] worktree add: be tolerant of corrupt worktrees
Quoted 30 lines · Duy Nguyen
Quoted 10 lines · Eric Sunshine
Quoted 4 lines · Nguyễn Thái Ngọc Duy
find_worktree()candie()unexpectedly because it usesreal_path()instead of the gentler version. When it's used in 'git worktree add' [1] and there's a bad worktree, thisdie()could prevent people from adding new worktrees.This is good to know because, to fix [1], I think we'll want to add a new function[2] akin to
find_worktree(), but without magic suffix matching (that is, just literal absolute path comparison).Yeah.
find_worktree()was made to handle command line options from worktree's move/remove, it's probably a bit too magical for this case.I still want to store relative path in "gitdir" files at some point, which would complicate the last "absolute path comparison" part a bit. But it should be manageable.
Quoted 10 lines
[1]: https://public-inbox.org/git/[email protected]/
[2]: https://public-inbox.org/git/CAPig+cQh8hxeoVjLHDKhAcZVQPpPT5v0AUY8gsL9=qfJ7z-L2A@mail.gmail.com/Quoted 6 lines
The "bad" condition to trigger this is when a parent of the worktree's location is deleted. Then
real_path()will complain.Use the other version so that bad worktrees won't affect 'worktree add'. The bad ones will eventually be pruned, we just have to tolerate them for a bit.
Quoted 6 lines
Quoted 4 lines
The patch itself makes sense, though, as Shaheed noted in his response, pruning seems to get short-circuited somehow under this situation; perhaps that needs its own fix, but certainly shouldn't hold up this fix.
I might have missed that detail. Thanks for pointing out. Will get another look.
On the off-chance this is not obvious to you experts, 'worktree remove' also hits this issue, as you can see from this little example I just caught:
$ git worktree remove --force /tmp/tmp56t59s2k/git_worktree.pa44kgs0 fatal: Invalid path '/tmp/tmpy4f98pwj': No such file or directory $ mkdir /tmp/tmpy4f98pwj $ git worktree remove --force /tmp/tmp56t59s2k/git_worktree.pa44kgs0 fatal: Invalid path '/tmp/tmp_q3p2mon': No such file or directory $ mkdir /tmp/tmp_q3p2mon ... several more suppressed ... $ git worktree remove --force /tmp/tmp56t59s2k/git_worktree.pa44kgs0 fatal: '/tmp/tmp56t59s2k/git_worktree.pa44kgs0' is not a working tree
Thanks, Shaheed
Quoted 3 lines
--
Duy