git clone with --separate-git-dir destroys existing directory content
cover git clone with --separate-git-dir destroys existing directory content
Hi,
I found an issue with git clone, when using --separate-git-dir and that directory already exists, it's content is destroyed.
I have created a patch set on origin/next, but (as I have learned this the hard way) please let me know if this needs to be backported...
Ben...
Ben Wijen (2):
git clone: check for non-empty directory
git clone: don't clone into non-empty directory
builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 3 ++- 2 files changed, 12 insertions(+), 3 deletions(-)
Signature · 1 line
2.27.0
PATCH 1/2 git clone: check for non-empty directory
When using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed.
Extend test to make sure content is left intact
--- t/t5601-clone.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
t/t5601-clone.sh +2 -1
@@ -271,7 +271,8 @@ test_expect_success 'fetch from gitfile parent' ' test_expect_success 'clone separate gitdir where target already exists' ' rm -rf dst && - test_must_fail git clone --separate-git-dir realgitdir src dst + test_must_fail git clone --separate-git-dir realgitdir src dst && + test -f realgitdir/config ' test_expect_success 'clone --reference from original' '
Signature · 1 line
2.27.0
Re: [PATCH 1/2] git clone: check for non-empty directory
Quoted 9 lines · Ben Wijen
When using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed.
Extend test to make sure content is left intact
Signed-off-by Ben Wijen <[email protected]>---
t/t5601-clone.sh test_expect_success 'fetch from gitfile parent' ' · line 275
Quoted 6 lines
t/t5601-clone.sh +2
-1@@ -271,7 +271,8 @@ test_expect_success 'fetch from gitfile parent' ' test_expect_success 'clone separate gitdir where target already exists' ' rm -rf dst && - test_must_fail git clone --separate-git-dir realgitdir src dst + test_must_fail git clone --separate-git-dir realgitdir src dst && + test -f realgitdir/config '
This addresses the immediate problem of the directory content being destroyed, which is good. But, we can future-proof it even more by also verifying (to some degree) that the existing content has not been disturbed. Doing so would give us greater confidence against some future change breaking "realgitdir" in some fashion other than merely emptying it. For instance, we might do this:
rm -rf dst &&
echo foo=bar >realgitdir/config &&
test_must_fail git clone --separate-git-dir realgitdir src dst &&
grep foo=bar realgitdir/configOne other observation: To preserve bisectability (git-bisect) of git.git itself, we want to ensure that the entire test suite still passes at each point in a patch series. Making this change to the test in its own patch introduces a failure into the test suite, which is undesirable. One way to address this shortcoming would be to temporarily change this test from 'test_must_succeed' to 'test_must_fail', and then flip it back to 'test_must_succeed' in patch 2/2. A cleaner approach is simply to combine the two patches so that the fix and updated test are bundled together (which makes sense anyhow in this case since this patch is so short and need not stand on its own).
Re: [PATCH 1/2] git clone: check for non-empty directory
Quoted 3 lines · Eric Sunshine
[...] For instance, we might do this:
rm -rf dst && echo foo=bar >realgitdir/config &&
I meant to use '>>' here, not '>'.
Re: [PATCH 1/2] git clone: check for non-empty directory
In reply to Eric SunshineQuoted 8 lines · Eric Sunshine
One other observation: To preserve bisectability (git-bisect) of git.git itself, we want to ensure that the entire test suite still passes at each point in a patch series. Making this change to the test in its own patch introduces a failure into the test suite, which is undesirable. One way to address this shortcoming would be to temporarily change this test from '
test_must_succeed' to 'test_must_fail', and then flip it back to 'test_must_succeed' in patch 2/2. [...]
And another correction: I meant 'test_expect_success' and 'test_expect_failure', respectively.
PATCH 2/2 git clone: don't clone into non-empty directory
In reply to Ben WijenWhen using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed.
Make sure we don't clone into an existing non-empty directory.
--- builtin/clone.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-)
builtin/clone.c +10 -2
@@ -946,7 +946,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)int is_bundle =0 , is_local;const char *repo_name, *repo, *work_tree, *git_dir;char *path, *dir, *display_repo =NULL ; -int dest_exists; +int dest_exists, real_dest_exists =0 ;const struct ref *refs, *remote_head;const struct ref *remote_head_points_at;const struct ref *our_head_points_at;
@@ -1021,6 +1021,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)die (_ ("destination path '%s' already exists and is not " "an empty directory." ), dir); +if (real_git_dir) { + real_dest_exists =path_exists (real_git_dir); +if (real_dest_exists && !is_empty_dir (real_git_dir)) +die (_ ("destination path '%s' already exists and is not " +"an empty directory." ), real_git_dir); + } + +strbuf_addf (&reflog_msg,"clone: from %s" , display_repo ? display_repo : repo);free (display_repo);
@@ -1057,7 +1065,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }if (real_git_dir) { -if (path_exists (real_git_dir)) +if (real_dest_exists) junk_git_dir_flags |=REMOVE_DIR_KEEP_TOPLEVEL ; junk_git_dir = real_git_dir; }else {
Signature · 1 line
2.27.0
Re: [PATCH 2/2] git clone: don't clone into non-empty directory
Quoted 9 lines · Ben Wijen
When using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed.
Make sure we don't clone into an existing non-empty directory.
Signed-off-by Ben Wijen <[email protected]>---
builtin/clone.c int cmd_clone(int argc, const char **argv, const char *prefix) · line 1026
Quoted 6 lines
builtin/clone.c +3
-0@@ -1021,6 +1021,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)die (_ ("destination path '%s' already exists and is not " "an empty directory." ), dir); +if (real_git_dir) { + real_dest_exists =path_exists (real_git_dir); +if (real_dest_exists && !is_empty_dir (real_git_dir))
"git init --separate-git-dir=" checks only whether the path exists, and errors out if it does; it doesn't care whether the directory is empty or not. I'm wondering, therefore, if this check should be tightened to more closely align the behavior of the two commands. (Using the tighter semantic now doesn't prohibit loosening it in the future, whereas it's harder to tighten behavior which started out loose.)
builtin/clone.c int cmd_clone(int argc, const char **argv, const char *prefix) · line 1028
Quoted 2 lines
builtin/clone.c +2
-0+die (_ ("destination path '%s' already exists and is not " +"an empty directory." ), real_git_dir);
This message might be a bit clearer if "destination" is changed to "repository". (And drop the bit about not being empty -- if we go that route.)
Re: [PATCH 2/2] git clone: don't clone into non-empty directory
Quoted 10 lines · Eric Sunshine
On Wed, Jul 1, 2020 at 5:46 AM Ben Wijen <[email protected]> wrote:
"git init --separate-git-dir=" checks only whether the path exists, and errors out if it does; it doesn't care whether the directory is empty or not. I'm wondering, therefore, if this check should be tightened to more closely align the behavior of the two commands. (Using the tighter semantic now doesn't prohibit loosening it in the future, whereas it's harder to tighten behavior which started out loose.)
About git init, I did take a look at it, but saw 'exist_ok' is always true when called from 'cmd_init_db' (see builtin/init_db.c:654) This means git init always allows when git_dir/real_git_dir already exists, which I understand for the worktree, but I doubt if one wants this for the repository. (when --separate-git-dir is used)
Also, because 'exist_ok' is always true for git init and - with this patch - git clone checks the git_dir/real_git_dir before the 'init_db' call I'm wondering if the code in 'init_db' (builtin/init-db.c:394-400) can be removed.
Then, even if we do take that route (no 'is_empty_dir', only 'path_exists') we must also address junk_git_dir_flags, as REMOVE_DIR_KEEP_TOPLEVEL would invalidate a second git clone.
So, as this patch only fixes the problem at hand, IMHO separate patch-sets should be created for mentioned issues.
Ben...
Quoted 2 lines · Ben Wijen
I found an issue with git clone, when using --separate-git-dir and that directory already exists, it's content is destroyed.
Thanks for fixing this.
Quoted 2 lines
I have created a patch set on origin/next, but (as I have learned this the hard way) please let me know if this needs to be backported...
Does this patch series depend on certain topics in 'next'? If not, please base it on 'master' instead. This will allow it to graduate from 'seen' to 'next' and eventually 'master' without being held back by any other topics in flight. On the other hand, if it does depend upon some topic in 'next', then it is a good idea to specify which specific topics it needs. That way, your patches will be able to graduate after those specific topics graduate rather than having to wait until _all_ of 'next' graduates (which could take forever).
v2 cover git clone: don't clone into non-empty directory
Eric, thank you for your review.
With this update I have:
* rebased (and squashed) on master
* included your proposals for the test
* changed "destination path..." to "repository path..."
Ben...
Ben Wijen (1):
git clone: don't clone into non-empty directory
builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 4 +++- 2 files changed, 13 insertions(+), 3 deletions(-)
Signature · 1 line
2.27.0
PATCH v2 1/1 git clone: don't clone into non-empty directory
In reply to Eric SunshineWhen using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed.
So, make sure we don't clone into an existing non-empty directory.
--- builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 4 +++- 2 files changed, 13 insertions(+), 3 deletions(-)
builtin/clone.c +10 -2
@@ -946,7 +946,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)int is_bundle =0 , is_local;const char *repo_name, *repo, *work_tree, *git_dir;char *path, *dir, *display_repo =NULL ; -int dest_exists; +int dest_exists, real_dest_exists =0 ;const struct ref *refs, *remote_head;const struct ref *remote_head_points_at;const struct ref *our_head_points_at;
@@ -1021,6 +1021,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)die (_ ("destination path '%s' already exists and is not " "an empty directory." ), dir); +if (real_git_dir) { + real_dest_exists =path_exists (real_git_dir); +if (real_dest_exists && !is_empty_dir (real_git_dir)) +die (_ ("repository path '%s' already exists and is not " +"an empty directory." ), real_git_dir); + } + +strbuf_addf (&reflog_msg,"clone: from %s" , display_repo ? display_repo : repo);free (display_repo);
@@ -1057,7 +1065,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }if (real_git_dir) { -if (path_exists (real_git_dir)) +if (real_dest_exists) junk_git_dir_flags |=REMOVE_DIR_KEEP_TOPLEVEL ; junk_git_dir = real_git_dir; }else {
t/t5601-clone.sh +3 -1
@@ -271,7 +271,9 @@ test_expect_success 'fetch from gitfile parent' ' test_expect_success 'clone separate gitdir where target already exists' ' rm -rf dst && - test_must_fail git clone --separate-git-dir realgitdir src dst + echo foo=bar >>realgitdir/config && + test_must_fail git clone --separate-git-dir realgitdir src dst && + grep foo=bar realgitdir/config ' test_expect_success 'clone --reference from original' '
Signature · 1 line
2.27.0
Re: [PATCH v2 1/1] git clone: don't clone into non-empty directory
Quoted 4 lines · Ben Wijen
When using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed.
So, make sure we don't clone into an existing non-empty directory.
This came from d45420c1 (clone: do not clean up directories we didn't create, 2018-01-02), where we claimed
Because we know that the only directory we'll write into is an empty one, we can handle this case by just passing the KEEP_TOPLEVEL flag to our recursive delete (if we could write into populated directories, we'd have to keep track of what we wrote and what we did not, which would be much harder).
The assumed use case is that somebody created an empty directory for our use in a grandparent directory where we have no write permission and gave it to us, and we want to do a "git clone" into it while keeping that directory alive. But we didn't make sure the directory was not empty ourselves---we just assumed it to be empty.
And the originally envisioned use case does not have anything to do with the use of --separate-git-dir at all. So from that point of view, this change does not break the originally envisioned use case, so this probably is not regressing any existing and valid use case that may not involve "--separate-git-dir", but I'd rather see that the commit message explain these things to its readers to justify itself.
Thanks.
PATCH v3 1/1 git clone: don't clone into non-empty directory
When using git clone with --separate-git-dir realgitdir and realgitdir already exists, it's content is destroyed.
So, make sure we don't clone into an existing non-empty directory.
Note: #d45420c1: "clone: do not clean up directories we didn't create" assumed we always write into an empty directory, but missed the check for real_git_dir, this commit fixed it by adding the check.
--- builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 4 +++- 2 files changed, 13 insertions(+), 3 deletions(-)
builtin/clone.c +10 -2
@@ -946,7 +946,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)int is_bundle =0 , is_local;const char *repo_name, *repo, *work_tree, *git_dir;char *path, *dir, *display_repo =NULL ; -int dest_exists; +int dest_exists, real_dest_exists =0 ;const struct ref *refs, *remote_head;const struct ref *remote_head_points_at;const struct ref *our_head_points_at;
@@ -1021,6 +1021,14 @@ int cmd_clone(int argc, const char **argv, const char *prefix)die (_ ("destination path '%s' already exists and is not " "an empty directory." ), dir); +if (real_git_dir) { + real_dest_exists =path_exists (real_git_dir); +if (real_dest_exists && !is_empty_dir (real_git_dir)) +die (_ ("repository path '%s' already exists and is not " +"an empty directory." ), real_git_dir); + } + +strbuf_addf (&reflog_msg,"clone: from %s" , display_repo ? display_repo : repo);free (display_repo);
@@ -1057,7 +1065,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) }if (real_git_dir) { -if (path_exists (real_git_dir)) +if (real_dest_exists) junk_git_dir_flags |=REMOVE_DIR_KEEP_TOPLEVEL ; junk_git_dir = real_git_dir; }else {
t/t5601-clone.sh +3 -1
@@ -271,7 +271,9 @@ test_expect_success 'fetch from gitfile parent' ' test_expect_success 'clone separate gitdir where target already exists' ' rm -rf dst && - test_must_fail git clone --separate-git-dir realgitdir src dst + echo foo=bar >>realgitdir/config && + test_must_fail git clone --separate-git-dir realgitdir src dst && + grep foo=bar realgitdir/config ' test_expect_success 'clone --reference from original' '
Signature · 1 line
2.27.0
v3 cover git clone: don't clone into non-empty directory
In reply to Junio C HamanoHi Junio,
I added a small note to the commit message.
Please let me know if you require more.
Thank you,
Ben Wijen (1):
git clone: don't clone into non-empty directory
builtin/clone.c | 12 ++++++++++-- t/t5601-clone.sh | 4 +++- 2 files changed, 13 insertions(+), 3 deletions(-)
Signature · 1 line
2.27.0